Correlation of performance data and structure data from an information technology environment

ABSTRACT

The disclosed system and method acquire and store performance measurements relating to performance of a component in an information technology (IT) environment and log data produced by the IT environment, in association with corresponding time stamps. The disclosed system and method correlate at least one of the performance measurements with at least one of the portions of log data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.15/421,398 filed Jan. 31, 2017 and titled “Processing Of PerformanceData And Structure Data From An Information Technology Environment”which is itself a Continuation-in-Part of U.S. patent application Ser.No. 14/167,316 filed Jan. 29, 2014 and U.S. patent application Ser. No.14/801,721 filed Jul. 16, 2015, and now issued as U.S. Pat. No.9,754,395. U.S. patent application Ser. No. 14/167,316 is itself aContinuation-in-Part of U.S. Pat. Nos. 8,904,389; 8,683,467; 9,164,786;and 9,495,187 all filed Apr. 30, 2013. The '316 application also claimsthe benefit of U.S. Provisional Application No. 61/883,869 filed Sep.27, 2013 and U.S. Provisional Application No. 61/900,700 filed Nov. 6,2013. U.S. patent application Ser. No. 14/801,721 is a Continuation ofU.S. patent application Ser. No. 14/253,548 filed Apr. 15, 2014, and nowissued as U.S. Pat. No. 9,142,049 which is itself a Continuation-in-Partof U.S. patent application Ser. No. 14/167,316, filed Jan. 29, 2014 andU.S. Pat. Nos. 8,904,389; 8,683,467; 9,164,786; and 9,495,187 all filedApr. 30, 2013. The '548 application also claims the benefit of U.S.Provisional Application No. 61/883,869 filed Sep. 27, 2013, U.S.Provisional Application No. 61/900,700 filed Nov. 6, 2013, and U.S.Provisional Application No. 61/979,484 filed Apr. 14, 2014. The entirecontents of each of the foregoing applications are incorporated byreference herein.

TECHNICAL FIELD

The present disclosure relates generally to techniques for processinglog data and/or performance data relating to components in aninformation technology (IT) environment.

BACKGROUND

Along with the advancement in computing technology, users' expectationsof computational capabilities are similarly increasing. Users areconstantly seeking resources that can provide the ability to achieve acomputational result quickly and appropriately. Attending to users'requests is complicated by the fact that user projects vary in terms ofrequired processing power, memory allocation, software capabilities,rights licensing, etc. Recently, systems have been organized to includea plurality of virtual machines. Tasks can then be assigned to virtualmachines based on the task requirements, the machines' capabilities andthe system load. However, given the dynamic nature of assignments andthe many components in these systems, monitoring the systems'performance is difficult.

SUMMARY

In accordance with the teachings provided herein, systems and methodsfor monitoring a hypervisor system are provided. A hypervisor system cancoordinate operations of a set of virtual machines (VM) and/or hosts.Characterizing the overall operation of the system and/or operation ofvarious system components can be complicated by the coordinatedoperation of the system components and the potential architectureflexibility of the system.

According to some embodiments, an architecture of a hypervisor structureis represented to a reviewer, along with indications characterizing howwell individual components of the system are performing. In oneinstance, the architecture (which may be defined by an architectureprovider and flexible in its structure) is represented as a tree withindividual nodes corresponding to system components. For individual VMs,a performance number is calculated based on task completions and/orresource utilization of the VM, and a performance state is assigned tothe component based on the number and state criteria. For higher-levelcomponents (e.g., hosts, host clusters, and/or a Hypervisor), anotherperformance number is calculated based on the states of the underlyingcomponents. A performance state is assigned to the higher-levelcomponents using different state criteria and the respective performancenumber.

A reviewer is presented with a performance indicator (which can includea performance statistic or state) of one or more high-level components.At this point, lower level architecture and/or corresponding performanceindicators are hidden from the reviewer. The reviewer can then select acomponent and “drill down” into performance metrics of underlyingcomponents. That is, upon detecting a reviewer's selection of acomponent, low-level architecture beneath the selected component ispresented along with corresponding performance indicators.

In some instances, a performance event can be generated based on one ormore performance assessments. Each performance event can correspond toone or more specific hypervisor components and/or a Hypervisor ingeneral. Each performance event can include performance data for thecomponent(s) and/or Hypervisor, such as a performance metric (e.g., CPUusage), performance statistic or performance state. In some instances,performance is assessed using different types of assessments (e.g., CPUusage versus memory usage). Multiple types of performance data can berepresented in a single event or split across events.

A time stamp can be determined for each performance event. The timestamp can identify a time at which a performance was assessed. Theevents can then be stored in a time-series index, such that events arestored based on their time stamps. Subsequently, the index can be usedto generate a result responsive to a query. In one instance, uponreceiving a query, performance events with time stamps within a timeperiod associated with the query are first retrieved. A late-bindingschema is then applied to extract values of interest (e.g., identifiersof hypervisor components, or a type of performance). The values can thenbe used to identify query-responsive events (e.g., such that onlyperformance events for component #1 are further considered) or identifyvalues of interest (e.g., to determine a mode CPU usage).

Time stamped events can also be stored for other types of information.Events can identify tasks (e.g., collecting, storing, retrieving, and/orprocessing of big-data) assigned to and/or performed by hypervisorcomponents and/or data received and/or processed by a hypervisorcomponent. For example, a stream of data (e.g., log files, big data,machine data, and/or unstructured data) can be received from one or moredata sources. The data can be segmented, time stamped and stored as dataevents (e.g., including machine data, raw data and/or unstructured data)in a time-series index (e.g., a time-series data store). Thus, ratherthan extracting field values at an intake time and storing only thefield values, the index can retain the raw data or slightly processedversions thereof and extraction techniques can be applied at query time(e.g., by applying an iteratively revised schema).

While it can be advantageous to retain relatively unprocessed data, itwill be appreciated that data events can include any or all of thefollowing: (1) time stamped segments of raw data, unstructured data, ormachine data (or transformed versions of such data); (2) the kinds ofevents analyzed by vendors in the Security Information and EventManagement (“SIEM”) field; (3) any other logical piece of data (such asa sensor reading) that was generated at or corresponds to a fixed time(thereby enabling association of that piece of data with a time stamprepresenting the fixed time); and (4) occurrences where some combinationof one or more of any of the foregoing types of events either meetsspecified criteria or was manually selected by a data analyst as notableor a cause for an alert.

Data events can be used to generate a response to a received query.Select data events (e.g., matching a time period and/or field constraintin a query) can be retrieved. A defined or learned schema can be appliedto extract field values from the retrieved events, which can beprocessed to generate a statistical query result (e.g., a count orunique identification) and/or selection (e.g., selecting events withparticular field values). A query event can include information from thequery and/or from the result and can also be time stamped and indexed.

Thus, one or more time-series indices can store a variety of timestamped events. This can allow a reviewer to correlate (e.g., based on amanual sampling or larger scale automated process) poor performancecharacteristics with processing tasks (e.g., data being indexed).

Particular embodiments of the subject matter described in thisspecification can be implemented to realize one or more of the followingadvantages. Techniques disclosed herein provide for the capability tocharacterize an operation of a hypervisor system at a variety of levels.By presenting the performance in a top-down manner, a reviewer canidentify a level at which a system is experiencing problems and how anarchitecture may be modified to alleviate the problems. Further, byclassifying different types of performance metrics (for various levelsin the hierarchy) into one of a same set of states, a reviewer caneasily understand how each portion of the system is performing.

The details of one or more embodiments of the disclosure are set forthin the accompanying drawings and the description below. Other features,aspects, and advantages of the disclosure will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 shows a block diagram of an embodiment of a virtual-machineinteraction system;

FIG. 2 shows a block diagram of an embodiment of task assigner;

FIG. 3 shows a block diagram of an embodiment of a VM monitoring system;

FIG. 4 illustrates an example of a representation of an architecture fora Hypervisor;

FIGS. 5A-5B illustrate an example of sequential presentations conveyingan architecture and system performance that can be presented to areviewer;

FIGS. 6A-6C illustrate example detailed information that can bepresented to characterize performance of a hypervisor system, a host anda VM, respectively;

FIGS. 7A-7C further illustrate example detailed information that can bepresented to characterize performance of a hypervisor system, a host anda VM, respectively;

FIG. 8 illustrates a flowchart of an embodiment of a process for using aVM machine to complete user tasks;

FIG. 9A illustrates a flowchart of an embodiment of a process forcharacterizing VM-system components' performance;

FIG. 9B illustrates a flowchart of an embodiment of a process forgenerating and using time stamped events to establish structurecharacteristics associated with a performance level;

FIG. 10 illustrates a flowchart of an embodiment of a process forassigning a performance state to a low-level component in a Hypervisor;

FIG. 11 illustrates a flowchart of an embodiment of a process forassigning a performance state to a high-level component in a Hypervisor;

FIG. 12 illustrates a flowchart of an embodiment of a process for usinga VM machine to complete user tasks;

FIG. 13 illustrates a flowchart of an embodiment of a process foranalyzing the performance of a Hypervisor using historical data;

FIG. 14 shows a block diagram of an embodiment of a data intake andquery system;

FIG. 15 illustrates a flowchart of an embodiment of a process forstoring collected data;

FIG. 16 illustrates a flowchart of an embodiment of a process forgenerating a query result;

FIG. 17 illustrates a flowchart of an embodiment of a process for usingintermediate information summaries to accelerate generation a queryresult;

FIG. 18 illustrates a flowchart of an embodiment of a process fordisplaying performance measurements and log data over a selected timerange;

FIGS. 19A-19F illustrate examples of ways to select a time range forretrieving performance measurements and log data;

FIGS. 20A-20B illustrate examples of detailed performance measurementsand log data that can be presented; and

FIG. 21 illustrates an example of a presentation of log data that isassociated with performance measurements.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

The ensuing description provides preferred exemplary embodiment(s) onlyand is not intended to limit the scope, applicability or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodiment.It is understood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

Each of the following U.S. patent applications is incorporated byreference herein in its entirety: U.S. patent application Ser. No.13/874,423; U.S. patent application Ser. No. 13/874,434; U.S. patentapplication Ser. No. 13/874,441; U.S. patent application Ser. No.13/874,448; U.S. patent application Ser. No. 14/167,316; and U.S. patentapplication Ser. No. 14/801,721. Additionally, each of U.S. provisionalpatent applications Nos. 61/883,869 and 61/900,700 is incorporated byreference herein in its entirety.

Referring first to FIG. 1, a block diagram of an embodiment of avirtual-machine interaction system 100 is shown. An architectureprovider 105, user 115 and/or performance reviewer 125 can interact witha task scheduler 140 and/or virtual-machine (VM) monitoring system 155via respective devices 110, 120 and/or 130 and a network 135, such asthe Internet, a wide area network (WAN), local area network (LAN) orother backbone. In some embodiments, VM monitoring system 155 is madeavailable to one or more of architecture provider 105, user 115 and/orperformance reviewer 125 via an app (that can be downloaded to andexecuted on a respective portable electronic device) or a website. Itwill be understood that, although only one of each of an architectureprovider 105, a user 115 and/or a performance reviewer 125 is shown,system 100 can include multiple architecture providers 105, users 115and/or performance reviewers 125.

Architecture-provider device 110, user device 120 and/or reviewer device130 can each be a single electronic device, such as a hand-heldelectronic device (e.g., a smartphone). It will be understood thatarchitecture-provider device 110, user device 120 and/or reviewer device130 can also include a system that includes multiple devices and/orcomponents. The device(s) 110, 120 and/or 130 can comprise a computer,such as the desktop computer, a laptop computer or a tablet. In someinstances, a provider 105, user 115 and/or performance reviewer 125 usesdifferent devices at different times to interact with task scheduler 140and/or VM monitoring system 155.

An architecture provider 105 can communicate with VM monitoring system155 to provide input defining at least part of an architecture that setsforth a structure of a Hypervisor. The input can include identificationof components of the Hypervisor, such as VMs, hosts or host clusters.The input can also include identification of relationships betweensystem components, which can include parent-child relationships. Forexample, a host can be identified as being a parent of five specificVMs. In some instances, identifying the relationships includes defininga hierarchy.

Architecture provider 105 can identify characteristics of particularhypervisor components, such as a CPU count, CPU type, memory size,operating system, name, an address, an identifier, a physical locationand/or available software. The architecture can also identifyrestrictions and/or rules applicable to VM-system components. Forexample, select resources may be reserved such that they can only beassigned high-priority tasks or tasks from particular users. As anotherexample, architecture provider 105 can identify that particularresources are only to be assigned tasks of a particular type or that alltasks of a particular type are to be assigned to a particular resource.

The input can include text entered into a field, an uploaded file,arrangement and/or selection of visual icons, etc. Defining thearchitecture can include defining a new structure or modifying anexisting structure.

Based on the architecture, a task scheduler 140 can utilize a set ofhosts 145 and/or VMs 150 to complete computational tasks. In someinstances, task scheduler 140 assigns tasks to a host 145 and/or VM 150(e.g., the host providing computing resources that support the VMoperation and the VM being an independent instance of an operatingsystem (“OS”) and software). The VM can then, e.g., store data, performprocessing and/or generate data. As described in further detail herein,task assignments can include collecting data (e.g., log files, machinedata, or unstructured data) from one or more sources, segmenting thedata into discrete data events, time stamping the data events, storingdata events into a time-series data store, retrieving particular dataevents (e.g., responsive to a query), and/or extracting values of fieldsfrom data events or otherwise processing events. Task scheduler 140 canmonitor loads on various system components and adjust assignmentsaccordingly. Further, the assignments can be identified to be inaccordance with applicable rules and/or restrictions.

A VM monitoring system 155 can monitor applicable architecture, taskassignments, task-performance characteristics and resource states. Forexample, VM monitoring system 155 can monitor: task completion time, apercentage of assigned tasks that were completed, a resource powerstate, a CPU usage, a memory usage and/or network usage. VM monitoringsystem 155 can use these monitored performance metrics to determineperformance indicators (as described further below) to present to areviewer 125. Reviewer 125 can interact with an interface provided by VMmonitoring system 155 to control which performance indicators arepresented. For example, reviewer 125 can specify a type of performanceindicator (e.g., by defining a set of performance states) or can specifyspecific components, component types or levels for which the indicatorsare presented.

As used in this disclosure, a “performance metric” may refer to acategory of some type of performance being measured or tracked for acomponent (e.g., a virtual center, cluster, host, or virtual machine) inan IT environment, and a “performance measurement” may refer to aparticular measurement or determination of performance at a particulartime for that performance metric.

Performance metrics may include a CPU performance metric, a memoryperformance metric, a summary performance metric, a performance metricbased on a max CPU usage, a performance metric based on a max memoryusage, a performance metric based on a ballooned memory, a performancemetric based on a swapped memory, a performance metric based on anaverage memory usage percentage, a performance metric based on the totalamount of memory that is reclaimed from all of the VMs on a host, aperformance metric based on the total amount of memory that is beingswapped from all of the VMs on a host, a performance metric that changesstate based on the remaining disk space on a data store, a performancemetric that changes state based on how much space is over-provisioned(i.e., negative numbers are a representation of an under-provisioneddata store), a performance metric based on a VM's average CPU usage inpercent, a performance metric based on a VM's average memory usage inpercent, a performance metric based on a VM's state waiting for CPUtime, a performance metric based on a VM's memory that is actively inuse, a performance metric based on a VM's memory saved by memorysharing, a performance metric based on a VM's memory used to power theVM, a performance metric based on physical memory that is mapped to a VM(i.e., memory not including overhead memory), a performance metric basedon an amount of physical memory that is being reclaimed by a hostthrough a ballooning driver, a performance metric based on memory thatis being read by a VM from a host's swap file, a performance metricbased on an amount of memory a VM has had to write to a swap file, aperformance metric based on an amount of memory from a VM that has beenswapped by a host. This is a host swapping and is always a sign of thehost being in stress, a performance metric based on an average read rateto virtual disks attached, a performance metric based on an averagewrite rate to virtual disks attached, a performance metric based on anaverage input/output (I/O) rate to a virtual disk, a performance metricbased on a number of times a VM wrote to its virtual disk, a performancemetric based on a number of times a VM read from its virtual disk, aperformance metric based on a time taken to process a SCSI command by aVM, a performance metric based on a number of commands that were abortedon a VM, a performance metric based on a number of SCSI-bus resetcommands that were issued, a performance metric based on an averageamount of bytes read across a VM's virtual network interface card (NIC),a performance metric based on an average amount of bytes broadcastedacross a VM's virtual NIC, a performance metric based on a combinedbroadcast and received rates across all virtual NIC instances, aperformance metric based on an average usage of a host's CPU in percent,a performance metric based on an amount of time a host waited for CPUcycles, a performance metric based on an average usage of a host's CPUin percent, a performance metric based on an average amount of allmemory in active state by all VMs and Virtual Provisioning X Daemonservices, a performance metric based on an average amount of memorybeing consumed by a host, which includes all VMs and an overhead of a VMkernel, a performance metric based on an average overhead of all VMs andan overhead of a vSphere, a performance metric based on an averagememory granted to all VMs and vSphere, a performance metric based on asum of all VM's memory control values for all powered-on VM, aperformance metric based on a combined sum of all swap-in values for allpowered-on VMs, a performance metric based on a combined sum of allswap-off values for all powered-on VMs, a performance metric based on anamount of memory from all VMs that has been swapped by a host, aperformance metric based on an average amount of bytes read from eachlogical unit number (LUN) on a host, a performance metric based on anaverage amount of bytes written to each LUN on a host, a performancemetric based on an average aggregated disk I/O for all VMs running on ahost, a performance metric based on a total number of writes to a targetLUN, a performance metric based on a total number of reads from a targetLUN, a performance metric based on a sum of kernel requests to a device,a performance metric based on a sum of kernel requests spent in a queuestate, a performance metric based on a number of commands that wereaborted on a host, a performance metric based on a number of SmallComputers System Interface (SCSI) bus reset commands that were issued, aperformance metric based on an average amount of data received across ahost's physical adapter, a performance metric based on an average amountof data broadcasted across a host's physical adapter, a performancemetric based on a combined broadcast and received rates across allphysical NIC instances, a performance metric based on an amount of CPUresources a VM would use if there were no CPU contention or CPU limit, aperformance metric based on an aggregate amount of CPU resources all VMswould use if there were no CPU contention or CPU limit, a performancemetric based on a CPU usage, which is an amount of actively used virtualCPUs, a performance metric based on a CPU usage, which is an aggregateof CPU usage across all VMs on a host, and/or a performance metric basedon an average CPU usage percentage.

Included below is a non-exhaustive list of known performance metricsthat may be monitored by default by VM monitoring system 155.PercentHighCPUVm, PercentHighMemVm, PercentHighSumRdyVm, VMInvCpuMaxUsg,VMInvMemMaxUsg, PercentHighBalloonHosts, PercentHighSwapHosts,PercentHighCPUHosts, BalloonedMemory_MB, swappedMemory_MB,RemainingCapacity_GB, Overprovisioned_GB, p_average_cpu_usage_percent,p_average_mem_usage_percent, p_summation_cpu_ready_millisecond,p_average_mem_active_kiloBytes, p_average_mem_consumed_kiloBytes,p_average_mem_overhead_kiloBytes, p_average_mem_granted_kiloBytes,p_average_mem_vmmemctl_kiloBytes, p_average_mem_swapin_kiloBytes,p_average_mem_swapout_kiloBytes, p_average_mem_swapped_kiloBytes,p_average_disk_read_kiloBytesPerSecond,p_average_disk_write_kiloBytesPerSecond,p_average_disk_usage_kiloBytesPerSecond,p_summation_disk_numberWrite_number, p_summation_disk_numberRead_number,p_latest_disk_maxTotalLatency_millisecond,p_summation_disk_commandsAborted_number,p_summation_disk_busResets_number,p_average_net_received_kiloBytesPerSecond,p_average_net_transmitted_kiloBytesPerSecond,p_average_net_usage_kiloBytesPerSecond, p_average_cpu_usage_percent,p_summation_cpu_ready_millisecond, p_average_mem_usage_percent,p_average_mem_active_kiloBytes, p_average_mem_consumed_kiloBytes,p_average_mem_overhead_kiloBytes, p_average_mem_granted_kiloBytes,p_average_mem_vmmemctl_kiloBytes, p_average_mem_swapin_kiloBytes,p_average_mem_swapout_kiloBytes, p_average_mem_llSwapUsed_kiloBytes,p_average_disk_numberReadAveraged_number,p_average_disk_numberWriteAveraged_number,p_average_disk_usage_kiloBytesPerSecond,p_summation_disk_numberWrite_number, p_summation_disk_numberRead_number,p_latest_disk_maxTotalLatency_millisecond,p_average_disk_queueLatency_millisecond,p_summation_disk_commandsAborted_number,p_summation_disk_busResets_number,p_average_net_received_kiloBytesPerSecond,p_average_net_transmitted_kiloBytesPerSecond,p_average_net_usage_kiloBytesPerSecond, p_average_cpu_demand_megaHertz,p_average_cpu_demand_megaHertz, p_average_cpu_usagemhz_megaHertz,p_average_cpu_usagemhz_megaHertz and/or AvgUsg_pctPercentHighCPUVm,PercentHighMemVm, PercentHighSumRdyVm, VMInvCpuMaxUsg, VMInvMemMaxUsg,PercentHighBalloonHosts, PercentHighSwapHosts, PercentHighCPUHosts,BalloonedMemory_MB, swappedMemory_MB, RemainingCapacity_GB,Overprovisioned_GB, p_average_cpu_usage_percent,p_average_mem_usage_percent, p_summation_cpu_ready_millisecond,p_average_mem_active_kiloBytes, p_average_mem_consumed_kiloBytes,p_average_mem_overhead_kiloBytes, p_average_mem_granted_kiloBytes,p_average_mem_vmmemctl_kiloBytes, p_average_mem_swapin_kiloBytes,p_average_mem_swapout_kiloBytes, p_average_mem_swapped_kiloBytes,p_average_disk_read_kiloBytesPerSecond,p_average_disk_write_kiloBytesPerSecond,p_average_disk_usage_kiloBytesPerSecond,p_summation_disk_numberWrite_number, p_summation_disk_numberRead_number,p_latest_disk_maxTotalLatency_millisecond,p_summation_disk_commandsAborted_number,p_summation_disk_busResets_number,p_average_net_received_kiloBytesPerSecond,p_average_net_transmitted_kiloBytesPerSecond,p_average_net_usage_kiloBytesPerSecond, p_average_cpu_usage_percent,p_summation_cpu_ready_millisecond, p_average_mem_usage_percent,p_average_mem_active_kiloBytes, p_average_mem_consumed_kiloBytes,p_average_mem_overhead_kiloBytes, p_average_mem_granted_kiloBytes,p_average_mem_vmmemctl_kiloBytes, p_average_mem_swapin_kiloBytes,p_average_mem_swapout_kiloBytes, p_average_mem_llSwapUsed_kiloBytes,p_average_disk_numberReadAveraged_number,p_average_disk_numberWriteAveraged_number,p_average_disk_usage_kiloBytesPerSecond,p_summation_disk_numberWrite_number, p_summation_disk_numberRead_number,p_latest_disk_maxTotalLatency_millisecond,p_average_disk_queueLatency_millisecond,p_summation_disk_commandsAborted_number,p_summation_disk_busResets_number,p_average_net_received_kiloBytesPerSecond,p_average_net_transmitted_kiloBytesPerSecond,p_average_net_usage_kiloBytesPerSecond, p_average_cpu_demand_megaHertz,p_average_cpu_demand_megaHertz, p_average_cpu_usagemhz_megaHertz,p_average_cpu_usagemhz_megaHertz and/or AvgUsg_pct.

Of course any of the above listed performance metrics could also oralternatively be monitored and reported in any of: bytes, megaBytes,gigaBytes and/or any other byte or memory amount. Any performancemetrics could also or alternatively be monitored and reported in any of:hertz, megaHertz, gigaHertz and/or any hertz amount. Moreover, any ofthe performance metrics disclosed herein may be monitored and reportedin any of percentage, relative, and/or absolute values.

Other performance metrics that may be collected may include any type ofcluster performance metrics, such as:latest_clusterServices_cpufairness_number,average_clusterServices_effectivecpu_megaHertz,average_clusterServices_effectivemem_megaBytes,latest_clusterServices_failover_number and/orlatest_clusterServices_memfairness_number. Of course any performancemetrics could also be monitored and reported in any of: bytes,megaBytes, gigaBytes and/or any byte amount. Any performance metricscould also be in hertz, megaHertz, gigaHertz and/or any hertz amount.

CPU performance metrics that may be collected may include any of:average_cpu_capacity.contention_percent,average_cpu_capacity.demand_megaHertz,average_cpu_capacity.entitlement_megaHertz,average_cpu_capacity.provisioned_megaHertz,average_cpu_capacity.usage_megaHertz, none_cpu_coreUtilization_percent,average_cpu_coreUtilization_percent,maximum_cpu_coreUtilization_percent,minimum_cpu_coreUtilization_percent,average_cpu_corecount.contention_percent,average_cpu_corecount.provisioned_number,average_cpu_corecount.usage_number, summation_cpu_costop_millisecond,latest_cpu_cpuentitlement_megaHertz, average_cpu_demand_megaHertz,latest_cpu_entitlement_megaHertz, summation_cpu_idle_millisecond,average_cpu_latency_percent, summation_cpu_maxlimited_millisecond,summation_cpu_overlap_millisecond, summation_cpu_ready_millisecond,average_cpu_reservedCapacity_megaHertz, summation_cpu_run_millisecond,summation_cpu_swapwait_millisecond, summation_cpu_system_millisecond,average_cpu_totalCapacity_megaHertz, average_cpu_totalmhz_megaHertz,none_cpu_usage_percent, average_cpu_usage_percent,minimum_cpu_usage_percent, maximum_cpu_usage_percent,none_cpu_usagemhz_megaHertz, average_cpu_usagemhz_megaHertz,minimum_cpu_usagemhz_megaHertz, maximum_cpu_usagemhz_megaHertz,summation_cpu_used_millisecond, none_cpu_utilization_percent,average_cpu_utilization_percent, maximum_cpu_utilization_percent,minimum_cpu_utilization_percent and/or summation_cpu_wait_millisecond Ofcourse any performance metrics could also be monitored and reported inany of: hertz, megaHertz, gigaHertz and/or any hertz amount.

Database and data store performance metrics that may be collected mayinclude any of: summation_datastore_busResets_number,summation_datastore_commandsAborted_number,average_datastore_datastoreIops_number,latest_datastore_datastoreMaxQueueDepth_number,latest_datastore_datastoreNormalReadLatency_number,latest_datastore_datastoreNormalWriteLatency_number,latest_datastore_datastoreReadBytes_number,latest_datastore_datastoreReadIops_number,latest_datastore_datastoreReadLoadMetric_number,latest_datastore_datastoreReadOIO_number,latest_datastore_datastoreVMObservedLatency_number,latest_datastore_datastoreWriteBytes_number,latest_datastore_datastoreWriteIops_number,latest_datastore_datastoreWriteLoadMetric_number,latest_datastore_datastoreWriteOIO_number,latest_datastore_maxTotalLatency_millisecond,average_datastore_numberReadAveraged_number,average_datastore_numberWriteAveraged_number,average_datastore_read_kiloBytesPerSecond,average_datastore_siocActiveTimePercentage_percent,average_datastore_sizeNormalizedDatastoreLatency_microsecond,average_datastore_throughput.contention_millisecond,average_datastore_throughput.usage_kiloBytesPerSecond,average_datastore_totalReadLatency_millisecond,average_datastore_totalWriteLatency_millisecond and/oraverage_datastore_write_kiloBytesPerSecond. Of course any performancemetrics could also be monitored and reported in any of: bytes,megaBytes, gigaBytes and/or any byte amount.

Disk performance metrics that may be collected may include any of:summation_disk_busResets_number, latest_disk_capacity_kiloBytes,average_disk_capacity.contention_percent,average_disk_capacity.provisioned_kiloBytes,average_disk_capacity.usage_kiloBytes, summation_disk_commands_number,summation_disk_commandsAborted_number,average_disk_commandsAveraged_number, latest_disk_deltaused_kiloBytes,average_disk_deviceLatency_millisecond,average_disk_deviceReadLatency_millisecond,average_disk_deviceWriteLatency_millisecond,average_disk_kernelLatency_millisecond,average_disk_kernelReadLatency_millisecond,average_disk_kernelWriteLatency_millisecond,average_disk_maxQueueDepth_number,latest_disk_maxTotalLatency_millisecond,summation_disk_numberRead_number,average_disk_numberReadAveraged_number,summation_disk_numberWrite_number,average_disk_numberWriteAveraged_number,latest_disk_provisioned_kiloBytes,average_disk_queueLatency_millisecond,average_disk_queueReadLatency_millisecond,average_disk_queueWriteLatency_millisecond,average_disk_read_kiloBytesPerSecond,average_disk_scsiReservationCnflctsPct_percent,summation_disk_scsiReservationConflicts_number,average_disk_throughput.contention_millisecond,average_disk_throughput.usage_kiloBytesPerSecond,average_disk_totalLatency_millisecond,average_disk_totalReadLatency_millisecond,average_disk_totalWriteLatency_millisecond,latest_disk_unshared_kiloBytes, none_disk_usage_kiloBytesPerSecond,average_disk_usage_kiloBytesPerSecond,minimum_disk_usage_kiloBytesPerSecond,maximum_disk_usage_kiloBytesPerSecond, latest_disk_used_kiloBytes and/oraverage_disk_write_kiloBytesPerSecond. Of course any performance metricscould also be monitored and reported in any of: bytes, megaBytes,gigaBytes and/or any byte amount.

Host-based replication (“hbr”) performance metrics that may be collectedmay include any of: average_hbr_hbrNetRx_kiloBytesPerSecond,average_hbr_hbrNetTx_kiloBytesPerSecond and/oraverage_hbr_hbrNumVms_number. Of course any performance metrics couldalso be monitored and reported in any of: bytes, megaBytes, gigaBytesand/or any byte amount.

Management Agent performance metrics that may be collected may includeany of: average_managementAgent_cpuUsage_megaHertz,average_managementAgent_memUsed_kiloBytes,average_managementAgent_swapIn_kiloBytesPerSecond,average_managementAgent_swapOut_kiloBytesPerSecond and/oraverage_managementAgent_swapUsed_kiloBytes. Of course any performancemetrics could also be monitored and reported in any of: bytes,megaBytes, gigaBytes and/or any byte amount.

Memory performance metrics that may be collected may include any of:none_mem_active_kiloBytes, average_mem_active_kiloBytes,minimum_mem_active_kiloBytes, maximum_mem_active_kiloBytes,average_mem_activewrite_kiloBytes,average_mem_capacity.contention_percent,average_mem_capacity.entitlement_kiloBytes,average_mem_capacity.provisioned_kiloBytes,average_mem_capacity.usable_kiloBytes,average_mem_capacity.usage_kiloBytes,average_mem_capacity.usage.userworld_kiloBytes,average_mem_capacity.usage.vm_kiloBytes,average_mem_capacity.usage.vmOvrhd_kiloBytes,average_mem_capacity.usage.vmkOvrhd_kiloBytes,average_mem_compressed_kiloBytes,average_mem_compressionRate_kiloBytesPerSecond,none_mem_consumed_kiloBytes, average_mem_consumed_kiloBytes,minimum_mem_consumed_kiloBytes, maximum_mem_consumed_kiloBytes,average_mem_consumed.userworlds_kiloBytes,average_mem_consumed.vms_kiloBytes,average_mem_decompressionRate_kiloBytesPerSecond,average_mem_entitlement_kiloBytes, none_mem_granted_kiloBytes,average_mem_granted_kiloBytes, minimum_mem_granted_kiloBytes,maximum_mem_granted_kiloBytes, none_mem_heap_kiloBytes,average_mem_heap_kiloBytes, minimum_mem_heap_kiloBytes,maximum_mem_heap_kiloBytes, none_mem_heapfree_kiloBytes,average_mem_heapfree_kiloBytes, minimum_mem_heapfree_kiloBytes,maximum_mem_heapfree_kiloBytes, average_mem_latency_percent,none_mem_llSwapIn_kiloBytes, average_mem_llSwapIn_kiloBytes,maximum_mem_llSwapIn_kiloBytes, minimum_mem_llSwapIn_kiloBytes,average_mem_llSwapInRate_kiloBytesPerSecond,none_mem_llSwapOut_kiloBytes, average_mem_llSwapOut_kiloBytes,maximum_mem_llSwapOut_kiloBytes, minimum_mem_llSwapOut_kiloBytes,average_mem_llSwapOutRate_kiloBytesPerSecond,none_mem_llSwapUsed_kiloBytes, average_mem_llSwapUsed_kiloBytes,maximum_mem_llSwapUsed_kiloBytes, minimum_mem_llSwapUsed_kiloBytes,average_mem_lowfreethreshold_kiloBytes,latest_mem_mementitlement_megaBytes, none_mem_overhead_kiloBytes,average_mem_overhead_kiloBytes, minimum_mem_overhead_kiloBytes,maximum_mem_overhead_kiloBytes, average_mem_overheadMax_kiloBytes,average_mem_overheadTouched_kiloBytes,average_mem_reservedCapacity_megaBytes,average_mem_reservedCapacity.userworld_kiloBytes,average_mem_reservedCapacity.vm_kiloBytes,average_mem_reservedCapacity.vmOvhd_kiloBytes,average_mem_reservedCapacity.vmkOvrhd_kiloBytes,average_mem_reservedCapacityPctpercent, none_mem_shared_kiloBytes,average_mem_shared_kiloBytes, minimum_mem_shared_kiloBytes,maximum_mem_shared_kiloBytes, none_mem_sharedcommon_kiloBytes,average_mem_sharedcommon_kiloBytes, minimum_mem_sharedcommon_kiloBytes,maximum_mem_sharedcommon_kiloBytes, latest_mem_state_number,none_mem_swapIn_kiloBytes, average_mem_swapIn_kiloBytes,minimum_mem_swapIn_kiloBytes, maximum_mem_swapIn_kiloBytes,none_mem_swapOut_kiloBytes, average_mem_swapOut_kiloBytes,minimum_mem_swapOut_kiloBytes, maximum_mem_swapOut_kiloBytes,none_mem_swapin_kiloBytes, average_mem_swapin_kiloBytes,maximum_mem_swapin_kiloBytes, minimum_mem_swapin_kiloBytes,average_mem_swapinRate_kiloBytesPerSecond, none_mem_swapout_kiloBytes,average_mem_swapout_kiloBytes, maximum_mem_swapout_kiloBytes,minimum_mem_swapout_kiloBytes,average_mem_swapoutRate_kiloBytesPerSecond, none_mem_swapped_kiloBytes,average_mem_swapped_kiloBytes, minimum_mem_swapped_kiloBytes,maximum_mem_swapped_kiloBytes, none_mem_swaptarget_kiloBytes,average_mem_swaptarget_kiloBytes, minimum_mem_swaptarget_kiloBytes,maximum_mem_swaptarget_kiloBytes, none_mem_swapunreserved_kiloBytes,average_mem_swapunreserved_kiloBytes,minimum_mem_swapunreserved_kiloBytes,maximum_mem_swapunreserved_kiloBytes, none_mem_swapused_kiloBytes,average_mem_swapused_kiloBytes, minimum_mem_swapused_kiloBytes,maximum_mem_swapused_kiloBytes, none_mem_sysUsage_kiloBytes,average_mem_sysUsage_kiloBytes, maximum_mem_sysUsage_kiloBytes,minimum_mem_sysUsage_kiloBytes, average_mem_totalCapacity_megaBytes,average_mem_totalmb_megaBytes, none_mem_unreserved_kiloBytes,average_mem_unreserved_kiloBytes, minimum_mem_unreserved_kiloBytes,maximum_mem_unreserved_kiloBytes, none_mem_usage_percent,average_mem_usage_percent, minimum_mem_usage_percent,maximum_mem_usage_percent, none_mem_vmmemctl_kiloBytes,average_mem_vmmemctl_kiloBytes, minimum_mem_vmmemctl_kiloBytes,maximum_mem_vmmemctl_kiloBytes, none_mem_vmmemctltarget_kiloBytes,average_mem_vmmemctltarget_kiloBytes,minimum_mem_vmmemctltarget_kiloBytes,maximum_mem_vmmemctltarget_kiloBytes, none_mem_zero_kiloBytes,average_mem_zero_kiloBytes, minimum_mem_zero_kiloBytes,maximum_mem_zero_kiloBytes, latest_mem_zipSaved_kiloBytes and/orlatest_mem_zipped_kiloBytes. Of course any performance metrics couldalso be monitored and reported in any of: bytes, megaBytes, gigaBytesand/or any byte amount.

Network performance metrics that may be collected may include any of:summation_net_broadcastRx_number, summation_net_broadcastTx_number,average_net_bytesRx_kiloBytesPerSecond,average_net_bytesTx_kiloBytesPerSecond, summation_net_droppedRx_number,summation_net_droppedTx_number, summation_net_errorsRx_number,summation_net_errorsTx_number, summation_net_multicastRx_number,summation_net_multicastTx_number, summation_net_packetsRx_number,summation_net_packetsTx_number, average_net_received_kiloBytesPerSecond,summation_net_throughput.contention_number,average_net_throughput.packetsPerSec_number,average_net_throughput.provisioned_kiloBytesPerSecond,average_net_throughput.usable_kiloBytesPerSecond,average_net_throughput.usage_kiloBytesPerSecond,average_net_throughput.usage.ft_kiloBytesPerSecond,average_net_throughput.usage.hbr_kiloBytesPerSecond,average_net_throughput.usage.iscsi_kiloBytesPerSecond,average_net_throughput.usage.nfs_kiloBytesPerSecond,average_net_throughput.usage.vm_kiloBytesPerSecond,average_net_throughput.usage.vmotion_kiloBytesPerSecond,average_net_transmitted_kiloBytesPerSecond,summation_net_unknownProtos_number, none_net_usage_kiloBytesPerSecond,average_net_usage_kiloBytesPerSecond,minimum_net_usage_kiloBytesPerSecond and/ormaximum_net_usage_kiloBytesPerSecond. Of course any performance metricscould also be monitored and reported in any of: bytes, megaBytes,gigaBytes and/or any byte amount.

Power performance metrics that may be collected may include any of:average_power_capacity.usable_watt, average_power_capacity.usage_watt,average_power_capacity.usagePct_percent, summation_power_energy_joule,average_power_power_watt and/or average_power_powerCap_watt.

Rescpu performance metrics that may be collected may include any of:latest_rescpu_actav1_percent, latest_rescpu_actav15_percent,latest_rescpu_actav5_percent, latest_rescpu_actpk1_percent,latest_rescpu_actpk15_percent, latest_rescpu_actpk5_percent,latest_rescpu_maxLimited1_percent, latest_rescpu_maxLimited15_percent,latest_rescpu_maxLimited5_percent, latest_rescpu_runav1_percent,latest_rescpu_runav15_percent, latest_rescpu_runav5_percent,latest_rescpu_runpk1_percent, latest_rescpu_runpk15_percent,latest_rescpu_runpk5_percent, latest_rescpu_sampleCount_number and/orlatest_rescpu_samplePeriod_millisecond.

Storage Adapter performance metrics that may be collected may includeany of: average_storageAdapter_OIOsPct_percent,average_storageAdapter_commandsAveraged_number,latest_storageAdapter_maxTotalLatency_millisecond,average_storageAdapter_numberReadAveraged_number,average_storageAdapter_numberWriteAveraged_number,average_storageAdapter_outstandingIOs_number,average_storageAdapter_queueDepth_number,average_storageAdapter_queueLatency_millisecond,average_storageAdapter_queued_number,average_storageAdapter_read_kiloBytesPerSecond,average_storageAdapter_throughput.cont_millisecond,average_storageAdapter_throughput.usag_kiloBytesPerSecond,average_storageAdapter_totalReadLatency_millisecond,average_storageAdapter_totalWriteLatency_millisecond and/oraverage_storageAdapter_write_kiloBytesPerSecond. Of course anyperformance metrics could also be monitored and reported in any of:bytes, megaBytes, gigaBytes and/or any byte amount.

Storage path performance metrics that may be collected may include anyof: summation_storagePath_busResets_number,summation_storagePath_commandsAborted_number,average_storagePath_commandsAveraged_number,latest_storagePath_maxTotalLatency_millisecond,average_storagePath_numberReadAveraged_number,average_storagePath_numberWriteAveraged_number,average_storagePath_read_kiloBytesPerSecond,average_storagePath_throughput.cont_millisecond,average_storagePath_throughput.usage_kiloBytesPerSecond,average_storagePath_totalReadLatency_millisecond,average_storagePath_totalWriteLatency_millisecond and/oraverage_storagePath_write_kiloBytesPerSecond. Of course any performancemetrics could also be monitored and reported in any of: bytes,megaBytes, gigaBytes and/or any byte amount.

System performance metrics that may be collected may include any of:latest_sys_diskUsage_percent, summation_sys_heartbeat_number,latest_sys_osUptime_second, latest_sys_resourceCpuAct1_percent,latest_sys_resourceCpuAct5_percent,latest_sys_resourceCpuAllocMax_megaHertz,latest_sys_resourceCpuAllocMin_megaHertz,latest_sys_resourceCpuAllocShares_number,latest_sys_resourceCpuMaxLimited1_percent,latest_sys_resourceCpuMaxLimited5_percent,latest_sys_resourceCpuRun1_percent, latest_sys_resourceCpuRun5_percent,none_sys_resourceCpuUsage_megaHertz,average_sys_resourceCpuUsage_megaHertz,maximum_sys_resourceCpuUsage_megaHertz,minimum_sys_resourceCpuUsage_megaHertz,latest_sys_resourceMemAllocMax_kiloBytes,latest_sys_resourceMemAllocMin_kiloBytes,latest_sys_resourceMemAllocShares_number,latest_sys_resourceMemConsumed_kiloBytes,latest_sys_resourceMemCow_kiloBytes,latest_sys_resourceMemMapped_kiloBytes,latest_sys_resourceMemOverhead_kiloBytes,latest_sys_resourceMemShared_kiloBytes,latest_sys_resourceMemSwapped_kiloBytes,latest_sys_resourceMemTouched_kiloBytes,latest_sys_resourceMemZero_kiloBytes and/or latest sys uptime second. Ofcourse any performance metrics could also be monitored and reported inany of: bytes, megaBytes, gigaBytes and/or any byte amount.

Debug performance metrics that may be collected may include any of:maximum_vcDebugInfo_activationlatencystats_millisecond,minimum_vcDebugInfo_activationlatencystats_millisecond,summation_vcDebugInfo_activationlatencystats_millisecond,maximum_vcDebugInfo_activationstats_number,minimum_vcDebugInfo_activationstats_number,summation_vcDebugInfo_activationstats_number,maximum_vcDebugInfo_hostsynclatencystats_millisecond,minimum_vcDebugInfo_hostsynclatencystats_millisecond,summation_vcDebugInfo_hostsynclatencystats_millisecond,maximum_vcDebugInfo_hostsyncstats_number,minimum_vcDebugInfo_hostsyncstats_number,summation_vcDebugInfo_hostsyncstats_number,maximum_vcDebugInfo_inventorystats_number,minimum_vcDebugInfo_inventorystats_number,summation_vcDebugInfo_inventorystats_number,maximum_vcDebugInfo_lockstats_number,minimum_vcDebugInfo_lockstats_number,summation_vcDebugInfo_lockstats_number,maximum_vcDebugInfo_lrostats_number,minimum_vcDebugInfo_lrostats_number,summation_vcDebugInfo_lrostats_number,maximum_vcDebugInfo_miscstats_number,minimum_vcDebugInfo_miscstats_number,summation_vcDebugInfo_miscstats_number,maximum_vcDebugInfo_morefregstats_number,minimum_vcDebugInfo_morefregstats_number,summation_vcDebugInfo_morefregstats_number,maximum_vcDebugInfo_scoreboard_number,minimum_vcDebugInfo_scoreboard_number,summation_vcDebugInfo_scoreboard_number,maximum_vcDebugInfo_sessionstats_number,minimum_vcDebugInfo_sessionstats_number,summation_vcDebugInfo_sessionstats_number,maximum_vcDebugInfo_systemstats_number,minimum_vcDebugInfo_systemstats_number,summation_vcDebugInfo_systemstats_number,maximum_vcDebugInfo_vcservicestats_number,minimum_vcDebugInfo_vcservicestats_number and/orsummation_vcDebugInfo_vcservicestats_number.

Resource performance metrics that may be collected may include any of:average_vcResources_cpuqueuelength_number,average_vcResources_ctxswitchesrate_number,average_vcResources_diskqueuelength_number,average_vcResources_diskreadbytesrate_number,average_vcResources_diskreadsrate_number,average_vcResources_diskwritebytesrate_number,average_vcResources_diskwritesrate_number,average_vcResources_netqueuelength_number,average_vcResources_packetrate_number,average_vcResources_packetrecvrate_number,average_vcResources_packetsentrate_number,average_vcResources_pagefaultrate_number,average_vcResources_physicalmemusage_kiloBytes,average_vcResources_poolnonpagedbytes_kiloBytes,average_vcResources_poolpagedbytes_kiloBytes,average_vcResources_priviledgedcpuusage_percent,average_vcResources_processcpuusage_percent,average_vcResources_processhandles_number,average_vcResources_processthreads_number,average_vcResources_syscallsrate_number,average_vcResources_systemcpuusage_percent,average_vcResources_systemnetusage_percent,average_vcResources_systemthreads_number,average_vcResources_usercpuusage_percent and/oraverage_vcResources_virtualmemusage_kiloBytes. Of course any performancemetrics could also be monitored and reported in any of: bytes,megaBytes, gigaBytes and/or any byte amount.

Virtual disk performance metrics that may be collected may include anyof: summation_virtualDisk_busResets_number,summation_virtualDisk_commandsAborted_number,latest_virtualDisk_largeSeeks_number,latest_virtualDisk_mediumSeeks_number,average_virtualDisk_numberReadAveraged_number,average_virtualDisk_numberWriteAveraged_number,average_virtualDisk_read_kiloBytesPerSecond,latest_virtualDisk_readIOSize_number,latest_virtualDisk_readLatencyUS_microsecond,latest_virtualDisk_readLoadMetric_number,latest_virtualDisk_readOIO_number, latest_virtualDisk_smallSeeks_number,average_virtualDisk_throughput.cont_millisecond,average_virtualDisk_throughput.usage_kiloBytesPerSecond,average_virtualDisk_totalReadLatency_millisecond,average_virtualDisk_totalWriteLatency_millisecond,average_virtualDisk_write_kiloBytesPerSecond,latest_virtualDisk_writeIOSize_number,latest_virtualDisk_writeLatencyUS_microsecond,latest_virtualDisk_writeLoadMetric_number and/orlatest_virtualDisk_writeOIO_number. Of course any performance metricscould also be monitored and reported in any of: bytes, megaBytes,gigaBytes and/or any byte amount.

VM operation performance metrics that may be collected may include anyof: latest_vmop_numChangeDS_number, latest_vmop_numChangeHost_number,latest_vmop_numChangeHostDS_number, latest_vmop_numClone_number,latest_vmop_numCreate_number, latest_vmop_numDeploy_number,latest_vmop_numDestroy_number, latest_vmop_numPoweroff_number,latest_vmop_numPoweron_number, latest_vmop_numRebootGuest_number,latest_vmop_numReconfigure_number, latest_vmop_numRegister_number,latest_vmop_numReset_number, latest_vmop_numSVMotion_number,latest_vmop_numShutdownGuest_number, latest_vmop_numStandbyGuest_number,latest_vmop_numSuspend_number, latest_vmop_numUnregister_number and/orlatest_vmop_numVMotion_number.

In an embodiment of the disclosure, the IT environment performancemetrics for which performance measurements can be collected include anyof the published performance metrics that is known to be collected forIT systems and virtual-machine environments in software made andproduced by VMWare, Inc.; individual performance measurements atspecific times for these performance metrics may be made available bythe software producing the measurements (e.g. VMWare software) throughapplication programming interfaces (APIs) in the software producing themeasurements. In embodiments of the present disclosure, theseperformance measurements made by software in an IT or virtual-machineenvironment (e.g., VMWare software) may be constantly retrieved throughthe software's API and stored in persistent storage, either as events(in a manner as described later in this specification) or in some otherformat in which they can be persisted and retrieved through atime-correlated search (the correlation being the time at which theperformance measurements were made or the time to which the performancemeasurements correspond). These performance measurements couldalternatively be stored in any of the ways described herein by thesoftware producing them without making them available through an API orretrieving them through an API. While VMWare software has beenreferenced as a potential source of performance measurements in an IT orvirtual-machine environment, it should be recognized that suchperformance measurements could be produced or collected by softwareproduced by any company that is capable of providing such environmentsor measuring performance in such environments.

Referring next to FIG. 2, a block diagram of an embodiment of taskscheduler 140 is shown. Task scheduler 140 can be, in part or in itsentirety, in a cloud. Task scheduler 140 includes a user account engine205 that authenticates a user 115 attempting to access a Hypervisor.User account engine 205 can collect information about user 115 and storethe information in an account in a user-account data store 210. Theaccount can identify, e.g., a user's name, position, employer,subscription level, phone number, email, access level to the Hypervisorand/or login information (e.g., a username and password). Informationcan be automatically detected, provided by user 115, provided by anarchitecture provider 105 (e.g., to specify which users can have accessto a system defined by a provided architecture) and/or provided by areviewer 125 (e.g., who may be identifying employees within a company ororganization who are to be allowed to access the Hypervisor).

In some instances, user account engine 205 determines whether a user 105is authorized to access the system by requesting login information(e.g., a username and password) from user 115 and attempting to matchentered login information to that of an account stored in user-accountdata store 210. In some instances, user account engine 205 determineswhether user 115 is authorized by comparing automatically detectedproperties (e.g., an IP address and/or a characteristic of user device120) to comparable properties stored in an account. User account engine205 can further, in some instances, determine which Hypervisors and/orwhich hypervisor components user 115 is authorized to use (e.g., basedon a user-provided code or stored information identifying accesspermissions).

Authorized users can then be granted access to a task definer 215, whichreceives a task definition from user 115. User 115 can define a task by,e.g., uploading a program code, entering a program code, defining taskproperties (e.g., a processing to be done, a location of data to beprocessed, and/or a destination for processed data), or defining taskrestrictions or preferences (e.g., requirements of resources to be usedor task-completion deadlines). In some instances, defining a taskincludes uploading data to be processed. In some instances, a task isdefined by executing a code provided by user 115 and defining portionsof the codes (e.g., during specific iterations) as distinct tasks. Taskdefiner 215 can verify that the task definition is acceptable (e.g.,being of an appropriate format, having restrictions that can be met andbeing estimated to occupy an acceptable amount of resources). Thisverification can include fixed assessments and/or assessments that arespecific to user 115 or a user group.

Defined tasks, in some instances, relate to data collection processes.Task definer 215 can identify data to be collected based on user input(e.g., identifying a source, a type of data to be collected and/or atime period during which data is to be collected) or through othermeans. Task definer 215 can then define data-collection tasks. Each taskcan pertain to a portion of the overall data-collection process. Forexample, when data is to be continuously collected from multiplesources, task definer 215 can define individual tasks, each relating toa subset of the sources and each involving a defined time period. Thesetasks can be assigned to machines identified as forwarders.

Tasks can further or alternatively include parsing collected data intoindividual data events, identifying a time stamp for each data event(e.g., by extracting a time stamp from the data) and/or storing timestamped data events in a time-series data store. These efforts aredescribed in further detail below.

In some instances, task definer 215 defines tasks related to a query.The query can be received from a search engine via a search-engineinterface 217. The query can identify events of interest. The query canbe for one or more types of events, such as data events or performanceevents (e.g., searching for performance events with below-thresholdperformance values of a performance metric). The query may, e.g.,specify a time period, a keyword (present anywhere in the event) and/ora value of a field constraint (e.g., that a value for a “method” fieldbe “POST”). Task definer 215 can define one or more retrieval,field-extraction and/or processing tasks based on the query. Forexample, multiple retrieval tasks can be defined, each involving adifferent portion of the time period. Task definer 215 can also define atask to apply a schema so as to extract particular value of fields or atask to search for a keyword. Values extracted can be for fieldsidentified in the query and/or for other fields (e.g., each fielddefined in the schema). Those retrieved events with query-matchingvalues of fields or keywords can then be selected (e.g., for furtherprocessing or for a query response).

Task definer 215 can further define a task to process retrieved events.For example, a task can include counting a number of events meeting acriteria (e.g., set forth in the query or otherwise based on the query);identifying unique values of a field identified in a query; identifyinga statistical summary (e.g., average, standard deviation, median, etc.)of a value of a field identified in a query.

It will be appreciated that, while the retrieval, field extraction of avalue, and/or processing tasks are referred to separately, any two ormore of these tasks can be combined into a single task. Further, ininstances where different components act on different portions of dataretrieved for a given query, a task may include combining results of thetask actions.

Upon determining that the task definition is acceptable, task definer215 generates a queue entry. The queue entry can include an identifierof the task, a characteristic of the task (e.g., required resourcecapabilities, estimated processing time, and/or estimated memory use),an identification of user 115, a characteristic of user 115 (e.g., anemployer, a position, a level-of-service, or resources which can beused) and/or when the task was received. In some instances, the queueentry includes the task definition, while in other instances, the queueentry references a location (e.g., of and/or in another data store) ofthe task definition.

A prioritizer 225 can prioritize the task based on, e.g., acharacteristic of the task, a characteristic of user 115 and/or when thetask was received (e.g., where either new or old tasks are prioritized,depending on the embodiment). Prioritizer 225 can also or alternativelyprioritize the task based on global, company-specific or user-specificusage of part or all of Hypervisor. For example, if many queue itemsrequire that a processing VM be running Operating System (OS) #1 (and/orif few resources run the OS), prioritizer 225 may prioritize queue itemspermissive of or requiring a different OS being run. Similarly,prioritizations can depend on a current load on part or all of aHypervisor. For example, tasks that can be assigned to a VM currentlyhaving a small CPU usage can be assigned high priority. Thus, a loadmonitor 230 can communicate with prioritizer 225 to identify a load(e.g., a processing and/or memory load) on specific resources and/orspecific types of resources.

In some instances, a task is prioritized based on data involved in thetask. Collection, storage, retrieval and/or processing of valuable datacan be prioritized over other tasks or over other corresponding tasks.Prioritization can also be performed based on a source identification ordata. Prioritization can also be performed based on task types. Forexample, data-collection and event-storage tasks (e.g., intake tasks)may be prioritized over event-retrieval and event-processing tasks(e.g., query-response tasks).

Prioritizing a task can include assigning a score (e.g., a numeric orcategorical score) to the task, which may include identifying some tasksthat are “high” priority. Prioritizing a task can include ranking thetask relative to tasks. The prioritization of a task can be performedonce or it can be repeatedly performed (e.g., at regular intervals orupon having received a specific number of new tasks). The prioritizationcan be performed before, while or after a queue item identifying thetask is added to the queue. The queue item can then be generated ormodified to reflect the prioritization.

An assigner 235 can select a queue entry (defining a task) from queue220 and assign it to one or more resources (e.g., a host cluster, a hostand/or a VM). The selection can be based on a prioritization of queueentries in queue 220 (e.g., such that a highest priority task isselected). The selection can also or alternatively depend on real-timesystem loads. For example, load monitor 230 can identify to assigner 235that a particular VM recently completed a task or had low CPU usage.Assigner 235 can then select a queue entry identifying a task that canbe performed by the particular VM. The assignment can include apseudo-random element, depend on task requirements or preferences and/ordepend on loads of various system components. For example, assigner 235can determine that five VMs have a CPU usage below a threshold, candetermine that three of the five have capabilities aligned with a giventask, and can then assign the task to one of the three VMs based on apseudo-random selection between the three. The assignment can furtherand/or alternatively reflect which Hypervisors and/or system componentsa user from whom a task originated is allowed to access. Assigner 235can update queue 220 to reflect the fact that a task is/was assigned toidentify the assigned resource(s).

A task monitor 240 can then monitor performance of the tasks andoperation states (e.g., processing usage, CPU usage, etc.) of assignedresources. Task monitor 240 can update queue 220 reflect performanceand/or resource-operation states. In some instances, if a performancestate and/or resource-operation state is unsatisfactory (e.g., is notsufficiently progressing), assigner 235 can reassign the task.

Referring next to FIG. 3, a block diagram of an embodiment of VMmonitoring system 155 is shown. VM monitoring system 155 can be, in partor in its entirety, in a cloud. VM monitoring system 155 includes areviewer account engine 305, which authenticates a reviewer attemptingto access information characterizing performance of a Hypervisor.Reviewer account engine 305 can operate similarly to user account engine205. For example, reviewer account engine 305 can generate revieweraccounts stored in a reviewer-account data store 310 where the accountincludes information such as the reviewer's name, employer,level-of-service, which Hypervisors/components can be reviewed, a levelof permissible detail for reviews, and/or login information. Revieweraccount engine 305 can then determine whether detected orreviewer-entered information (e.g., login information) matchescorresponding information in an account.

VM monitoring system 155 also includes an activity monitor 315, whichmonitors activity of hypervisor components. The activity can include,for example, when tasks were assigned, whether tasks were completed,when tasks were completed, what tasks were assigned (e.g., requiredprocessing), users that requested the task performance, whether the taskwas a new task or transferred from another component (in which case asource component and/or transfer time can be included in the activity),CPU usage, memory usage, characteristics of any memory swapping orballooning (e.g., whether it occurred, when it occurred, an amount ofmemory, and the other component(s) involved), and/or any errors.

Activity monitor 315 can store the monitored activity (e.g., as or in anactivity record) in an activity data store 320. In one instance, one,more or each VM component is associated with a record. Performancemetrics of the component (e.g., CPU usage and/or memory usage) can bedetected at routine intervals. The record can then include an entry witha time stamp and performance metrics. Task assignments (including, e.g.,a time of assignment, a source user, whether the task was transferredfrom another component, a type of task, requirements of the task,whether the task was completed, and/or a time of completion) can also beadded to the record. In some instances, performance metrics are detected(and a corresponding record entry is generated and stored) upondetecting a task action (e.g., assignment, transfer, or completion)pertaining to the VM component. Thus, activity data store 320 canmaintain an indexed or organized set of metrics characterizinghistorical and/or current performance of hypervisor components.

An aggregator 325 can collect performance metrics from select activityrecords. The performance metrics can include, e.g., CPU usage, memoryusage, tasks assignments, task completions and/or any of the abovementioned performance metrics. The desired values of performance metricscan also include values generated from entries with time stamps within aparticular time period. In some instances, performance metrics arecollected from one or more entries having a most recent time stamp(e.g., a most recent entry or all entries within a most-recent 24-hourperiod).

The activity records can be selected based on an architecture stored inan architecture data store 330, the architecture defining a structure(e.g., components and component relationships) of a Hypervisor.Architectures can also specify which specific users or types of userscan use some or all of the Hypervisor and/or which specific reviewer ortypes of reviewers can access (some or all available) performanceindicators.

The architecture can be one provided by an architecture provider 105.For example, architecture provider 105 can interact with an architecturemanager 335 to define resources in a Hypervisor and relationshipsbetween components of the system. These definitions can be provided,e.g., by entering text, manipulating graphics or uploading a file. Itwill be appreciated that, while not shown, VM monitoring system 155 canfurther include an architecture-provider account engine andarchitecture-provider account data store that can be used toauthenticate an architecture provider. Architecture-provider accountscan include information similar to that in user accounts and/or revieweraccounts, and the architecture-provider account engine can authenticatean architecture provider in a manner similar to a user or reviewerauthentication technique as described herein.

FIG. 4 illustrates an example of a representation of an architecture fora Hypervisor. The depicted architecture is hierarchical and includes aplurality of nodes arranged in a plurality of levels. Each nodecorresponds to a component in the Hypervisor. The hierarchy defines aplurality of familial relationships. For example, VM 6 is a child ofHost 2 and a grandchild of the Host Cluster. The top level is thevirtual center where tasks are assigned. The second level is ahost-cluster level, which indicates which underlying hosts havetask-transferring arrangements with each other (the same-levelinteraction being represented by the dashed line). The third level is ahost level that provides computing resources that support VM operation.The fourth level is a VM level. Thus, based on the depictedarchitecture, an assignment to VM 7 would also entail an assignment toHost 2 and to the Host Cluster; an assignment to VM 3 would also entailan assignment to Host 1.

Returning to FIG. 3, aggregator 325 can aggregate performance metricsfrom records pertaining to a particular component in the architecture.As will be described in further detail below, performance indicators(determined based on performance metrics) associated with components atdifferent levels can be sequentially presented to a reviewer (e.g., in atop-down manner and responsive to reviewer selection of components).Thus, VM monitoring system 155 can, in some instances, also sequentiallydetermine performance indicators (determining lower level indicatorsfollowing a presentation of higher-level indicators and/or to reviewerselection of a component). VM monitoring system 155 can first determineperformance indicators for higher-level components and subsequently foreach of a subset or all of lower-level components. Thus, aggregator 325can first aggregate performance metrics in activity records for each ofone or more higher-level components and later aggregate performancemetrics in activity records for each of one or more lower-levelcomponents. It will be appreciated that other sequences can be utilized(e.g., repeatedly cycling through components in a sequence).

A statistics generator 340 can access the collection of performancemetrics and generate one or more performance statistics based on thevalues of one or more performance metrics. A performance statistic canpertain to any of the various types of performance metrics, such as aCPU usage, a memory usage, assigned tasks, a task-completion duration,etc. The statistic can include, e.g., an average, a median, a mode, avariance, a distribution characteristic (e.g., skew), a probability(which may be a percentage), a conditional probability (e.g.,conditioned on recent assignment of a task), a skew, and/or an outlierpresence. The statistic can include one or more numbers (e.g., an errorand a standard deviation). In some instances, the statistic includes aseries of numbers, such as histogram values. Statistics generator 340can store the statistic (in association with an identifier of arespective component and time period) in a statistics data store 345.Statistics generator 340 can identify which component and/or time periodare to be associated with the statistic based on what aggregation wasperformed.

A state engine 350 can access one or more state criteria fromstate-criteria data store 355 and use the state criteria and thegenerated statistic to assign a state (e.g., to a component and/or timeperiod). The state can then be stored (e.g., in association with arespective component and/or time period) in a state data store 360.State engine 350 can identify which component and/or time period are tobe associated with the state based on what aggregation was performed.

The state criteria can include one or more thresholds, a function and/oran if-statement. In one instance, two thresholds are set to define threestates: if a statistic is below the first threshold, then a first state(e.g., a “normal” state) is assigned; if a statistic is between thethresholds, then a second state (e.g., a “warning” state) is assigned;if a statistic is above the second threshold, then a third state (e.g.,a “critical state”) is assigned. The state criteria can pertain tomultiple statistics (e.g., having a function where a warning state isassigned if any of three statistics are below a respective threshold orif a score generated based on multiple statistics is below a threshold).

A state of a node corresponding to a component in an IT environment maybe based on performance measurements (corresponding to a performancemetric) made directly for that component, or it may depend on the statesof child nodes (corresponding to child components) of the node (e.g., awarning state if any of the child nodes are in a warning state, or awarning state if at least 50% of the child nodes are in a warningstate). A component in an IT environment may include a virtual center, acluster (of hosts), a host, or virtual machines running in a host, wherea cluster is a child component of a virtual center, a host is a childcomponent of a cluster, and a virtual machine is a child component of ahost.

The state criteria can include a time-sensitive criteria, such as athreshold based on a past statistic (e.g., indicating that a warningstate should be assigned if the statistic has increased by 10-20% sincea previous comparable statistic and a warning state should be assignedif it has increased by 20+%), a derivative (calculated based on acurrent and one or more past statistics) and/or an extrapolation(calculated based on a current and one or more past statistics).

In some instances, multiple states are defined. For example, an overallstate can be assigned to the component, and other specific statespertaining to more specific performance qualities (e.g., memory usage,processor usage and/or processing speed) can also be assigned.

The state criteria can be fixed or definable (e.g., by an architectureprovider 105 or reviewer 125). The state criteria can be the same acrossall components and/or time periods or they can vary. For example,criteria applicable to VM components can differ from criteria applicableto higher level components.

In some instances, the state criteria are determined based on aresults-oriented empirical analysis. That is, a state engine 350 can usean analysis or model to determine which values of performance metrics(e.g., a range of values) are indicative of poor or unsatisfactoryperformance of the Hypervisor. Thus, a result could be a performancemetric for a higher level component or a population user satisfactionrating.

An alarm engine 365 can access one or more alarm criteria fromalarm-criteria data store 370 and use the alarm criteria and an assignedstate to determine whether an alarm is to be presented. In one instance,an alarm criterion indicates that an alarm is to be presented if one ormore states are assigned. In one instance, an alarm criterion includes atime-sensitive assessment, such as a criterion that is satisfied whenthe state has changed to (or below) a specific state and/or has changedby a particular number of states since a last time point.

Alarm engine 365 can present the alarm by, e.g., presenting a warning onan interface (e.g., a webpage or app page), transmitting an email,sending a message (e.g., a text message), making a call or sending apage. A content of the alarm (e.g., email, message, etc.) can identify acurrent state and/or statistic, a previous state and/or statistic, atrend in the state and/or statistic, an applicable component, anapplicable time period, and/or an applicable Hypervisor.

VM monitoring system 155 can include an interface engine 375 thatenables a reviewer 115 to request a performance report and/or receive aperformance report. The report can include one or more statistics,states, and/or alarm statuses. The report can identify which componentand/or time period are associated with the statistic, state and/or alarmstatus. Interface engine 375 can present most-recent or substantiallyreal-time values (e.g., numerical statistics or states) and/orhistorical values. In some instances, interface engine accesses a set ofvalues for a given component, and generates and presents a table, list,or graph to illustrate a change in a performance. The report can alsoinclude activity pertaining to a component and/or time period (e.g.,tasks assigned, task statuses, etc.).

Interface engine 375 can receive input from reviewer 115, which cancause different information to be presented to the user. In someinstances, interface engine 375 merely accesses different data (e.g.,states, statistics, alarm statuses and/or activities) from data store320, 345, and/or 360. Interface engine 375 can then present the accesseddata itself or generate and present a representation of the data (e.g.,generate and present a graph). In some instances, the input causesinterface engine 375 to request that aggregator 325 aggregate differentperformance metrics, that statistics generator 340 generate differentstatistics, that state engine 350 generate different states and/or thatalarm engine 365 re-assess alarm criteria. The new data can then bepresented to reviewer 115. Thus, the report can be dynamic.

In some instances, the input can include selection of a component. Theselection can lead to a presentation (and potentially a generation of)more detailed data pertaining to the component and/or to a presentationof data pertaining to components that are children of the selectedcomponent. This former strategy can encourage a user to follow branchesdown an architecture tree to find, e.g., a source of a high-levelproblem or to understand best-performing branches.

While activity data store 320, statistics data store 345 and states datastore 360 are shown separately, it will be appreciated that two or moreof the data stores can be combined in a single data store. Each of one,more or all of the data stores can include a time-series data store. Inone instance, a performance event can be generated to identify one ormore of each of a value or values of a performance metric, statistic orstate. For example, a performance event can include a task-completionrate for a single VM over the past hour. A single event can be generatedto include performance values for an individual hypervisor component,performance values for each of multiple hypervisor components, orperformance values for each hypervisor component in a Hypervisor.

The performance event can identify one or more multiple components. Forexample, when a performance event includes performance values formultiple components, the performance event can identify the componentand/or other multiple components with particular familial relationships(e.g., parent, grandparent, child) to the component in a Hypervisorenvironment.

Each performance event can be time stamped or can otherwise beassociated with a time. The time stamp or time can indicate a time ortime period for which performance data identified in the event applies.Performance events (e.g., time stamped performance events) can be storedin one or more time-series data stores. Thus, select performance eventscorresponding to a time period of interest (of a reviewer) can beretrieved and analyzed

As described, statistics generator 340 can generate statistics and/orstate engine 350 can generate states based on collected values of one ormore performance metrics. In one instance, the statistic and/or stategeneration is performed in real-time subsequent to collection of values(i.e., performance measurements) of one or more performance metrics.Alternatively or additionally, statistics and/or states can bedetermined retrospectively. For example, time stamped performance eventscan include raw values for performance metrics. Periodically, or inresponse to receiving a query, performance events within a time periodbe retrieved and one or more statistics and/or one or more states can begenerated based on the retrieved events. This retrospective analysis canallow for dynamic definitions of states and/or statistics. For example,a reviewer can define a statistic to facilitate a particular outlierdetection or a reviewer can adjust a stringency of a “warning” state.

FIGS. 5A-5B illustrate an example of sequential presentations conveyingan architecture and system performance that can be presented to areviewer 125. In FIG. 5A, three relatively high-level nodes arepresented. Specifically a highest-level node is presented along with itschildren. In this instance, the children are at different levels inorder to ensure that each presented node has multiple children. It willbe appreciated that in other embodiments, the depicted children nodesare in the same level (e.g., such that another “Host Cluster” would be aparent of “Host 1” and have no other children).

As shown, this architecture includes 12 nodes that are hidden in therepresentation in FIG. 5A. The node hiding can help a user focus on amost likely lower-level cause of an overall sub-par performance.

An overall state of the represented components is indicated based onwhether the node is surrounded by a diamond. In this case, nodes in awarning state are surrounded by a diamond. It will be appreciated thatother state indicators (e.g., colors, text, icon presence or a number)can be used instead of or in addition to the surrounding indicator.

In this example, a reviewer 125 can select a node by clicking on it.FIG. 5B shows a representation of the architecture and systemperformance after reviewer 125 selected the Host 1 node (having awarning-state indicator). At this point, the children of Host 1 appear.Two of the child VM nodes also have a warning-state indicator.

FIG. 5B also illustrates how presentations can indicate which nodes areparent nodes. In this case, “fills” or patterns of the node convey thischaracteristic, with pattern nodes indicating that the nodes are notparents.

The structure-based and concise presentations shown in FIGS. 5A and 5Ballow a reviewer to drill down into sub-optimal system performance, toeasily understand which system components are properly operating and toeasily understand architecture underlying a Hypervisor. However, moredetailed performance information can also be presented to a reviewer.For example, detailed information can appear as a transient pop-up whena reviewer 125 hovers a cursor over a component and/or can appear as areport when a reviewer 125 double clicks on a node.

In some instances, an architecture provider 105 and reviewer 125 are asame party. The reviewer 125 can then review a representation, such asone shown in FIGS. 5A-5B and access performance indicators of specificsystem components. In the same-party instances, reviewer 125 can use thesame representation to modify an architecture. For example, reviewer 125can add, move or delete connections, move child components, add and/orremove components. Reviewer 125 can also select a particular component(e.g., by double clicking a node) and change its properties.

FIGS. 6A-6C illustrate example detailed information that can bepresented to characterize performance of a Hypervisor, a host and a VM,respectively. These graphics can be presented in response to a reviewer125 hovering over a specific hypervisor component. FIG. 6A shows gaugespresenting information pertaining to an overall Hypervisor. The gaugesidentify a percentage of VMs in a Hypervisor having undesirable states.The left gauge shows a percentage of VMs with a state for CPU usage in a“high” category. The middle gauge shows a percentage of VMs with a statefor memory usage in a “high” category. The right gauge shows apercentage of VMs within a state for an amount of time a VM is waitingto use a processor that is in a “high” category. Thus, 33% of VMs areseemingly affected in their processing capabilities based on overloadingof 2% of VMs. Thus, it would be useful to identify which VMs are withinthe 2% and/or 4.2% and a source of the problem for those VMs.

It will be appreciated that other high-level performance indicators canbe presented (e.g., ones related to memory. For example, other gaugescould identify memory performance indicators. For example, a gauge couldidentify a percentage of hosts with a “high” amount of memory beingused, having a “high” amount of memory ballooning (during which a hostis requesting memory be returned from a VM to the host), or having a“high” amount of memory swapping (during which a host is forcefullytaking back memory from a VM). Host processing characteristics (e.g., apercentage of hosts with “high” CPU usage) can also be presented forhosts.

These same gauges could be associated with a node representation of anIT system component (e.g., a node representing a virtual center, cluster(of hosts), a host, or a virtual machine) to indicate a performancemeasurement (relative to a maximum for the corresponding metric) for thecomponent or to indicate the percentage of child components of thecomponent that are in various states. In such an embodiment, the gaugecould partially surround the representation of the node, sitting (e.g.)just above the representation of the node. Where the gauge shows statesof child component, each color of the gauge takes up a percentage of thegauge corresponding to the percentage of child components having a statecorresponding to the color.

FIG. 6B shows information pertaining to a particular host in aHypervisor. The presented data compares performance characteristics ofthe host's children to more global comparable characteristics. The leftbar graph shows a histogram across VMs assigned to the host identifyinga sum-ready performance metric (identifying a time that the VM must waitbefore using a processor). The right bar graph is comparable butcharacterizes all VMs within a Hypervisor. In this instance, the righthistogram is highly skewed to the left, while the left histogram doesnot exhibit a similar skew. The histogram thus suggests that thesub-network of the host and its children is not operating as well as ispossible.

FIG. 6C shows a time-graph of the same waiting-time metrics for a VMacross period of times (in the lighter line). Specifically, each pointin the graph represents the performance value of waiting-time metricsfor a period of time. A comparable average for the performance values ofthe waiting-time metrics across all VMs is simultaneously presented (inthe darker line). The higher values underscore sub-optimal performance,as the processor is experiencing higher than average wait times. Thispresentation allows a reviewer 125 to understand whether a VM'sperformance is particularly poor relative to other VMs' performances, toidentify whether and when any substantial changes in the performanceoccurred, and to identify and when poor performance is becoming aconsistent problem. Further, the historical plot may allow a reviewer125 to notice a positive or negative trend in the values of one or moreperformance metrics, such that a problem can be remedied before itbecomes serious.

The historical presentation in FIG. 6C thus offers valuable insight asto a component's performance, when a change in performance occurred, andwhether the performance warrants a change in the VM architecture. Thehistorical presentation, however, requires that historical performancecharacteristics be stored and indexed (e.g., by time and/or component).This is complicated by the fact that this can be a very large amount ofdata. Storing all raw values of performance metrics involves not onlystoring a very large amount of data, but also repeatedly re-aggregatingthe values of the performance metrics and repeatedly recalculating thehistorical performance statistics and/or states. This can result in adelay of a presentation to a reviewer 125, which can be particularlynoticeable if the presentation is supposed to be presented transientlyand quickly as the reviewer hovers his cursor over a particulardepiction. Meanwhile, storing only statistics and/or states and not thevalues of the performance metrics limits the ability to customize whichstatistics and/or states are presented (e.g., by fixing time periodsinstead of allowing statistics to be calculated on a flexible basisdepending on a reviewer's interest and reviewing time) and can itselfeven lead to a large amount of data to store, due to many types ofperformance variables being calculated at many levels (meaning that asingle value of a performance metric may, in combination with othervalues of performance metrics, give rise to several values ofperformance statistics and/or states).

FIGS. 7A-7C further illustrate example detailed information that can bepresented to characterize the performance of a Hypervisor, a host and aVM, respectively. These reports can be presented in response to areviewer 125 selecting (e.g., by double clicking) a specific VM-systemcomponent. FIG. 7A illustrates a report for a Hypervisor. The report caninclude information about hosts in the system and VMs in the system. Thereport can identify system properties, such as a number and type ofcomponents within the system. In the illustrated example, the systemincludes 4 hosts and 74 VMs. The report can also characterizeprovider-initiated or automatic architecture changes, such as a numberof times a VM automatically migrated to another host (e.g., based on ahost-clustering architecture defined by an architecture provider). Itwill be appreciated that more and/or more detailed information can bepresented regarding architecture changes, such as identifying whetherthe change was automatic, identifying a time of the change, and/oridentifying involved components.

In this example, a host-status section identifies hosts by name andstorage capacity. A current status of each host is also indicated byshowing an amount of the host's capacity that is committed to serve VMsand an amount by which the host is overprovisioned. High commitment andoverprovisioning numbers can be indicative of poor performance. It willbe appreciated that the host information could be expanded to include,e.g., an overall or host-specific memory-ballooning or memory-swappingstatistic, host-clustering arrangements, and/or an overall orhost-specific CPU usage.

The report can also identify past alarms in an alarm-history section.For each alarm, an applicable component can be identified, a time of thealarm can be identified and a substance or meaning of an alarm can beidentified. These alarms can identify state changes for particularcomponents.

FIG. 7B illustrates a report for a host. Overall performance statisticsand corresponding states are presented in a host-statistics section.These statistics can be recent or real-time statistics and can beequivalent to instantaneous values of one or more performance metrics orcan be calculated using values of one or more performance metrics from arecent time period. A host-configurations section identifies theequipment and capabilities of the host. A connected-datastores sectionidentifies which other hosts in the Hypervisor the instant host isconnected to (e.g., via a clustering arrangement). In some instances,the section is expanded to identify a type of connection or a length oftime that the connection has existed.

A VM-information section identifies VMs assigned to the host. In theillustrated example, the report identified a number of VMs that areassigned and a number of those in a power-on state. The report alsoidentifies the number of VMs that migrated to or from the host (e.g.,via a host-clustering arrangements). The report can list recent VMtasks, events and/or log entries, and can identify an applicable time,VM and description. For example, tasks can include changing a resourceconfiguration for a VM, adding a VM to a host, and establishing a remoteconnection. Events can include presented alarms, VM migrations (fromhost to host), task migrations (from VM to VM), and warnings potentialarchitecture problems (e.g., based on actual or predicted insufficiencyof resources to support assigned child components or tasks). Log entriescan include identifications of unrecognized URI versions and softwarewarnings.

A historical-host-performance section shows how a performance statistichas been changing over time. In the depicted instance, the historicalstatistics (which can include a final real-time statistic) are showngraphically, along with a “normal” threshold (shown as the bottom, darkdashed line) and a “critical” threshold (shown as the top, gray dashedline). Reviewer 125 is able to set settings to control the statisticalpresentation. For example, reviewer 125 can identify a performancemetric of interest (e.g., CPU usage, memory usage, etc.), whether datais to be aggregated across VMs to derive the statistic, a statistic type(e.g., average, median, maximum, minimum, mode, variance, etc.), and atime period (e.g., 24 hours). Other settings may further be presented,such as time discretization during the time period and graph-formattingoptions (e.g., marker presence, marker size, line style, axis-ticksettings, etc.).

FIG. 7C illustrates a report for a VM. A VM-configurations sectionidentifies the resources allocated to the VM and other VM and/orrelationship characteristics (e.g., a name, assigned host and/orassigned cluster). A connected-datastores section identifies which hostsare, per an existing architecture, responsible for providing resourcesto the VM. A configuration-change-history section identifies a time andtype of a past change to the configuration of the VM and a partyinitiating the change.

A migration-request-history identifies any attempts and/or successes formigrating the VM from one host to the next. Thus, in this case, itappears as though the VM was attempting to migrate off of the host butfailed. This report also includes a historical-performance section,which can have similar presentation and setting-changing abilities asthe similar section from the host report. It will be appreciated that,e.g., thresholds can differ between the two. For example, a warningthreshold can be stricter for a host, since more VMs contribute to thestatistic and diminish the probability of observing extreme values.

It will also be appreciated that reports can include links to otherreports. For example, in the report in FIG. 7C, a reviewer 125 can clickon “Host1” to move to the report shown in FIG. 7B for that component.Thus, reviewer 125 can navigate via the reports to access performanceand configuration details for related hypervisor components.

Thus, the presentations shown from FIGS. 5A-7C show a variety of ways bywhich a reviewer 125 can understand how a Hypervisor is structured andperforming. By tying together structural and performance information, areviewer 125 can begin to understand what architecture elements may begiving rise to performance problems and can appropriately improve thearchitecture. Further, the presentations show how a given performancemeasure compares to other performance measures. One such comparison isan inter-system-component comparison, which can enable a reviewer 125 toidentify a reasonableness of a performance metric and determine a levelat which a problem could best be addressed. Another such comparison is ahistorical comparison, which can allow reviewer 125 to identifyconcerning trends and/or to pinpoint times at which substantialperformance changes occurred. Reviewer 125 can then reviewconfiguration-change or task histories to determine whether any eventslikely gave rise to the performance change.

It will be appreciated that alternative detailed information can bepresented to characterize performance of a hypervisor component. Thedetailed information can identify information about particular tasks ortypes of tasks assigned to the component. The information can includeevents related to the tasks. For example, a reviewer 125 can click on acomponent assigned to index data (or a component above the indexingcomponent in a hierarchy), and information about the events (e.g., anumber of events, unique field values, etc.) and/or the eventsthemselves can be presented. In one instance, clicking on a componentcan include a list of recently performed tasks. A reviewer 125 canselect an event-defining and storing task, and a number of the storedevents can be presented. Upon a further selection or automatically(e.g., subsequently or simultaneously), details (e.g., field valuesand/or time stamps) of the events can be presented, and/or the fullevents can be presented.

As noted herein, initial indexing tasks can create events derived fromraw data, unstructured data, semi-structured data, and/or machine data(or slightly transformed versions thereof) to be stored in data stores.This storage technique can allow a reviewer to deeply investigatepotential causes for poor performance. For example, a reviewer may beable to hypothesize that a component's poor performance is likely due toa type of task performed (e.g., extracting fields from events withinconsistent patterns or needing to index events without a time stampincluded therein).

FIG. 8 illustrates a flowchart of an embodiment of a process 800 forusing a VM machine to complete user tasks. Process 800 begins at block810, where task definer 215 defines a task. The task can be definedbased on user input, a data-collection effort and/or a query. In oneinstance, input is received (e.g., from a user) that is indicative of arequest to collect data (e.g., once or repeatedly). Task definer 215 canthen define one or more tasks to collect the data. When more than onetask is defined, they may be simultaneously defined or defined atdifferent times (e.g., the times relating to collection periodsidentified in the request). For any given collection effort, in someinstances, task definer 215 can parse the collection intosub-collections (e.g., each associated with a different portion of acollection time period), and a different task can be defined for eachsub-collection.

In one instance, task definer 215 defines data-segment and storagetasks, which may be defined as data is collected or otherwise received.In one instance, task definer 215 defines one or more retrieval and/orprocessing tasks in response to receiving a query or determining that aquery-response time is approaching. For example, a query may request aresponse at routine intervals, and tasks can be defined and performed inpreparation for each interval's end. The query can be one defined by anauthenticated user.

Prioritizer 225 prioritizes the task request (e.g., based oncharacteristics of user 110, characteristics of the task, system loadand/or when the request was received) at block 815. The prioritizationcan include generating a score, assigning a priority class or assigninga ranking. Task definer 215 places a queue item identifying the task inqueue 220 at block 820. The priority of the task can be reflected withinthe queue item itself, by the queue item's placement within a ranking orby a priority indicator associated with the queue item. Load monitor 230monitors loads of virtual machines (e.g., and/or hosts) at block 825.The monitoring can include detecting characteristics of tasks beingprocessed (e.g., resource requirements, a current total processing time,and/or which user who submitted the task). Assigner 235 selects the taskfrom queue 220 at block 830. The selection can occur, e.g., once thetask is at sufficiently high priority to be selected over other tasksand can further occur once appropriate resources are available toprocess the task. Assigner 235 assigns the task to a VM at block 835.The VM to which the task is assigned can be a VM with sufficientavailable resources to process the task. Assignment to a VM can furtherinclude assigning the task to a host and/or host cluster.

Task monitor 240 monitors performance of the task at the assigned VM atblock 840. For example, task monitor 240 can detect whether a VM appearsto be stalled in that it has not completed the task for over a thresholdduration of time. As another example, task monitor 240 can monitor howmuch of the VM's processing power and/or memory appears to be beingconsumed by the task performance. As another example, task monitor 240can determine whether any errors are occurring during the taskperformance. In some instances, task monitor 240 determines that theperformance is unsatisfactory at block 845 (e.g., based on too muchconsumption of the VM resources, too long of a processing time and/ortoo many errors), and assigner subsequently reassigns the task to adifferent VM at block 850. The different VM can be one with moreresources than the initial VM, one in a larger host-clustering network,and/or one currently processing fewer or less intensive tasks ascompared to those otherwise being processed by the initial VM.

In some instances, process 800 can further include generation andstorage of individual task events. A task event can identify informationdefining a task, an identification of when a task was assigned (orreassigned) an identification of a VM to which the task was assignedand/or a performance characteristic for the task (e.g., a start and/orstop processing time, a processing-time duration and/or whether anyerrors occurred). The task event can be time stamped (e.g., with a timethat the event was created, a time that task processing began orcompleted or an error time) and stored in a time-series data store.

FIG. 9A illustrates a flowchart of an embodiment of a process 900 forcharacterizing hypervisor components' performance. Process 900 begins atblock 905, where activity monitor 315 monitors performance of VMs andhosts. Through this monitoring, activity monitor 315 can detect valuesof performance metrics, such as CPU usage, memory usage, task assignmentcounts, task assignment types, task completion counts, and/or migrationsto/from the VM or to/from the host. Activity monitor 315 stores thedetected values of performance metrics in activity data store 320 atblock 910.

Aggregator 325 accesses an applicable architecture from architecturedata store 330 at block 915. The applicable architecture can be oneassociated with a reviewer, one randomly selected, or one defining aHypervisor of interest. The architecture can identify some or all of theVMs and/or hosts monitored at block 905. The architecture can identifyrelationships from the VM to other hypervisor components.

Aggregator 325 identifies one of the components from the architectureand a time period. The time period can include a current time/timeperiod (i.e., real-time or most recent time in activity data store 320for the component) or a previous time period. In some instances, process900 first characterizes performance of low-level components (e.g., VMs)before characterizing performance of high-level components.

Aggregator 325 accesses appropriate values of one or more performancemetrics or states at block 920. In some instances, for low-levelcomponents, values of one or more performance metrics can be accessedfrom activity data store 320. In some instances, for high-levelcomponents, states of children of the components can be accessed fromstate data store 360. In some instances, values of one or moreperformance metrics are accessed from activity data store 320 for allcomponents.

Statistics generator 340 generates a statistic based on the accessedmetrics or states and stores the statistic in statistic data store 345at block 925. The statistic can include, e.g., an average or extrememetric across the time period or a percentage of children componentshaving been assigned to one or more specific states (e.g., any of statesred, orange, or yellow).

State engine 350 accesses one or more state criteria from state-criteriadata store 355 at block 930. Which state criteria are accessed candepend on which component is being assessed. In one instance, differentlevels in an architecture have different criteria.

State engine 350 assesses the criteria in view of the statistic todetermine which state the component is in during the time period. Stateengine 350 then assigns the component to that state (as a present stateor a past state associated with the time period) at block 935.

State engine 350 stores the state in association with the component andtime period in state data store 360 at block 940. Process 900 can thenreturn to block 920 and repeat blocks 920-940 for a different componentand/or a different time period. For example, process can repeat in thismanner to continue to identify and store current statistics and/orstates.

It will be appreciated that values of one or more performance metrics,one or more statistics and/or one or more states can be stored in atime-series data store. In one instance, one or more events are createdand stored. Each event can include one or more performance-datavariables (e.g., values of performance metric, statistic and/or state)and an identifier of a hypervisor component corresponding to theperformance-data variable(s). A single event can correspond to a singlehypervisor component or multiple hypervisor components.

Each event can include or can otherwise be associated with a time stamp.In one instance, the time stamp corresponds to the performance-datavariable(s) (e.g., indicating when performance was monitored). Eachevent can then be stored in a bucket in a data store that corresponds to(e.g., includes) the time stamp. This storage technique can facilitatesubsequent time-based searching.

FIG. 9B illustrates a flowchart of an embodiment of a process 950 forgenerating and using time stamped events to establish structurecharacteristics associated with strong performance. Process 950 beginsat block 955, where a structure or architecture of aninformation-technology environment (e.g., a Hypervisor environment) ismonitored. The monitoring can include determining a number of componentswithin the environment, a number of a particular type of component(e.g., VMs, hosts or clusters in the environment), and/or relationshipsbetween components in the environment (e.g., identifying which VMs areassigned to which hosts or identifying other parent-childrelationships). This monitoring can, in some instances, be accomplishedby detecting each change (e.g., initiated based on input from anarchitecture provider) made to the structure.

At block 960, a time stamped event identifying a characteristic of thestructure can be identified. The event can identify, e.g., one or moreparent-child relationships and/or a number of total components (and/orcomponents of a given type) in the environment. In one instance, theevent identifies a portion or all of a hierarchy of the environment. Thetime stamp can be set to a time at which the characteristic was present(e.g., a time at which the structure was monitored at block 905). In oneinstance, multiple events include information characterizing anenvironment operating at a given timepoint (e.g., each even pertainingto a different component operating in the environment and identifyingany parent and/or child component in a hierarchy). One or more generatedstructure events can be stored in a time-series data store at block 965(e.g., by storing the event in a bucket including the time stamp of theevent).

At block 970, performance of each of one, more or all components in theenvironment can be monitored. For example, values of one or moreperformance metrics can be monitored for VMs and/or hosts. In someinstances, a performance statistic and/or state are generated based onthe monitored metrics.

A time stamped performance event can be generated at block 975. Theevent can identify performance data (e.g., one or more values ofmetrics, statistics and/or states) for one or more components in theenvironment (e.g., and identifiers of the one or more components). Thetime stamp for the event can identify a time for which the performancedata was accurate (e.g., a time of monitoring giving rise to theperformance data). One or more performance events can be stored in atime-series data store at block 980. The time-series data store at whichthe performance events are stored can be the same as or different fromthe performance events at which the structure events are stored (e.g.,by storing the event in a bucket including the time stamp of the event).

At block 985, performance characteristics can be correlated withcharacteristics of the information-technology (IT) environment. In oneinstance, a set of performance events and a set of structure events,each set corresponding to a time period, are retrieved from thetime-series data store(s). Each of one or more performance events can beassociated with structure characteristics of an information technologyenvironment operating at that time. For example, a structure event witha time stamp most recently preceding a time stamp of a performance eventcan identify the structure.

After the events are retrieved, information from the events can beextracted from the events (e.g., using a late-binding schema). Theinformation that is extracted can include performance data, componentidentifiers and/or structure information (e.g., parent-childrelationships and/or components present in an environment).

A high-level statistic can be determined based on performance data for aset of components. For example, the high-level statistic can include anextrema (e.g., indicative of a worst or best performance), a mean, amedian, a mode, a standard deviation or a range. The high-levelstatistic can be defined based on a fixed definition and/or input (e.g.,such that a reviewer can define a high-level statistic of interest). Astructure characteristic (which can be numeric) can also be determinedbased on extracted structure information. The structure characteristiccan include, e.g., a number of total components (e.g., hosts and VMs) inan environment (e.g., Hypervisor environment); a number of a given typeof components (e.g., a number of hosts or clusters) in the environment;and/or an average, median, minimum or maximum number of children of aparticular type of parent (e.g., a maximum number of VMs supported by asingle host or an average number of hosts assigned to a given cluster).In some instances, structure events identify changes in structure (e.g.,addition of VM). In these instances, determining a structurecharacteristic can include modifying a previous characteristic (e.g., toidentify a previous VM count and add one to the count).

Thus, a set of high-level statistics, each associated with a time, canbe determined. For each statistic, a corresponding structurecharacteristic can be identified (e.g., by identifying a structurecharacteristic associated with a time most recent to a time associatedwith the high-level statistic; or by identifying a structurecharacteristic associated with a time matching a time associated withthe high-level statistic). Thus, a matching set of structurecharacteristics can be identified. The set of high-level statistics andthe set of structure characteristics can be analyzed (e.g., using acorrelation analysis or model) to estimate influence of structurecharacteristics on performance

For example, using a set of structure events, a set of VMs supported bya particular host can be identified for multiple timepoints.Corresponding performance events can then be used to establish arelationship between a number of VMs assigned to the host and a “worst”performance statistic from amongst the set of VMs. As another example, adetermination can be made as to whether assigning two hosts to a singlecluster improved an average performance of the two hosts as compared toindependent operation of the hosts. This determination can be performedby using performance and structure events to identify, for eachtimepoint in a set of timepoints, a performance metric for the hosts andwhether the hosts were assigned to a cluster.

One or more performance events, structure events, performance data (orhigh-level performance statistics), structure characteristics, and/orcorrelation results can be presented to a reviewer. For example,structure characteristics identified as being correlated to poor orstrong performance can be identified to a user, or a relationshipbetween a characteristic and performance can be identified.

It will be appreciated that the performance influence of structurecharacteristics can be investigated using alternative techniques. Forexample, changes (e.g., improvements or degradations) in high-levelperformance statistics can be detected and structure changes precedingthe changes can be identified. As another example, changes in structurecharacteristics can be detected, and subsequent high-level performancestatistics can be identified. Averages, weighted on a type or magnitudeof performance or structure change can be used to evaluate influence.

State determinations for higher-level components can depend on directperformance measurements for a performance metric for the higher-levelcomponent, or it may depend on performances of underlying childrenlow-level components. One technique for arriving at the higher-levelstate would then be to aggregate performance metrics from all childrencomponents, generate a statistic based on the aggregated metrics, andidentify a state based on the statistic. However, this approach couldlead to a positive state assignment even in the case where a smallnumber of children components were performing very poorly. The aggregateanalysis could over-look this problem due to the mitigation of the poordata by other positive data from properly performing childrencomponents. Thus, another approach is to first identify a state for eachchild component, and then to determine a state for a parent componentbased on the states (not the direct metrics) of the child components.The state criteria can then set forth, e.g., a threshold number of childstate assignments to a negative state that would cause the parentcomponent also to be assigned to a negative state. FIGS. 10-11illustrate example processes for state assignments determined using thisapproach.

FIG. 10 illustrates a flowchart of an embodiment of a process 1000 forassigning a performance state to a low-level component in a Hypervisor.Process 1000 begins at block 1005, where aggregator 325 accesses anapplicable architecture from architecture data store 330. Thearchitecture identifies a particular VM, and aggregator 325 accessesvalues of one or more performance metrics characterizing the VM'sperformance during a time period from activity data store 320 at block1010. Based on the values of one or more performance metrics, statisticgenerator 340 generates a performance statistic (e.g., an average of themetrics) at block 1015.

State engine 350 accesses one or more state criteria from state-criteriadata store 355 at block 1020. In some instances, state-criteria datastore 355 includes multiple criteria, which may apply to differentcomponent types (e.g., having different configurations or capabilities),different architecture levels, different architectures, and/or differentreviewers. Thus, at block 1020, state engine 350 can select the criteriathat are applicable to the VM and/or to a reviewing reviewer. Stateengine 350 evaluates the statistic in view of the accessed criteria,and, as a result of the evaluation, assigns a state to the VM at block1020.

FIG. 11 illustrates a flowchart of an embodiment of a process 1100 forassigning a performance state to a high-level component in a Hypervisor.Process 1100 begins at block 1105, where aggregator 325 accesses anapplicable architecture from architecture data store 330. Thisarchitecture can be the same architecture as accessed at block 1005 inprocess 1000. The architecture can include a component that is a parentof the VM from process 1000. Thus, the architecture can include aVM-group component (e.g., a host).

Aggregator 325 accesses a state, from state data store 360, for each VMin the VM group at block 1110. Statistics generator 340 generates aperformance statistic based on the accessed states at block 1115. Thestatistic can include, e.g., an average, a percentage of VMs beingassigned to a particular state, a percentage of VMs being assigned to aparticular state or a worse state, etc. State engine 350 accesses statecriteria from state-criteria data store 355 at block 1120. As in process1000, this access can include selecting the criteria that are applicableto the VM group and/or reviewing reviewer. It will be appreciated thatthe state criteria accessed at block 1120 can differ from the statecriteria accessed at block 1020. State engine 350 evaluates thestatistic in view of the accessed criteria, and, as a result of theevaluation, assigns state to VM group at block 1120.

Despite the potential difference in the criteria used in processes 1000and 1100, the types of potential states that can be assigned can besimilar or the same. This can enable a reviewer 125 to easily understandhow well the component is performing without having to understand thedifferent criteria used in the assessment.

FIG. 12 illustrates a flowchart of an embodiment of a process 1200 forusing a VM machine to complete user tasks. Process 1200 begins at block1205, where reviewer account engine 305 authenticates a reviewer 125.

At block 1210, interface engine 375 presents, to reviewer 125, a dynamicrepresentation of at least part of an architecture of a Hypervisor and,for each of a set of components represented in the architecture, aperformance state assigned to the component. In some instances, thearchitecture and performance states are simultaneously represented toreviewer 125.

The architecture can be presented by displaying a series of nodes—eachnode representing a hypervisor component. The nodes can be connected toshow relationships. Relationships can include, e.g., resource-providingrelationships (e.g., between a host and VM), migration-enablingrelationships (e.g., between two hosts in a cluster, which can bedenoted via a direct connection or an indirect connection via an upperlevel host-cluster component). The nodes can be presented in ahierarchical manner, and relationships can include familial (e.g.,parent-child) relationships. It will be appreciated that thearchitecture can be presented in a variety of other manners. Forexample, a series of lists can identify, for each of a set ofcomponents, respective “children” components. As another example, rowsand columns in a matrix can identify columns, and cells in the matrixcan identify relationship presence and/or a type of relationship.

The presentation of the architecture can include identifying allcomponents and relationships in the architecture or a subset of thecomponents and relationships. The subset can include, e.g., componentsin a highest level in the architecture or in the highest n levels (e.g.,n being 2, 3, 4, etc.) and not components in the lower levels. Such arepresentation can encourage a reviewer 125 to assess a Hypervisor'sperformance in a top-down manner, rather than requiring that a reviewer125 already know a lower-level source of sub-optimal performance.

A performance state can be represented by a color, word, pattern, icon,or line width. For example, nodes in a representation of an architecturecan have an appearance characteristic (e.g., a line color, a linethickness, or a shading) that depends on the state of the representedcomponent.

The performance state can include an overall performance state. Theoverall performance state can be determined based on a plurality offactors, such as CPU usage, memory usage, task-processing times,task-processing intake numbers, and/or received or transmitted taskmigrations. In some instances, a value for each factor is identified andweighted, and a sum of the weighted values is used to determine theoverall performance state. In some instances, an overall performancestate depends on whether any of one or more factors fail respectivesatisfaction criteria or fall into a particular state (e.g., a warningstate).

In some instances, the performance state is not an overall performancestate but instead relates to a particular performance factors. Statespertaining to different performance factors can be simultaneouslypresented (e.g., via matrices or lists or via repeated presentation of afamily tree with state distinguishers). In one instance, a single familytree is shown to represent the architecture, and each node can have agraphical element (e.g., a line width, line color, shading, iconpresence, etc.) that represents a state for one performance factor.Thus, e.g., by looking at line width, a reviewer 125 could evaluateCPU-usage performances, and, by looking at line color, reviewer 125could evaluate memory-usage performances.

In some instances, a reviewer 125 can select a performance factor ofinterest. For example, a user can select “CPU usage” from aperformance-factor menu, and nodes in a family tree can then bedifferentially represented based on their CPU-usage performance.

Interface engine 375 detects a selection from reviewer 125 of a firstarchitecture component at block 1215. The selection can include, e.g.,clicking on or hovering over a component representation (e.g., a node,column heading, or row heading).

Interface engine 375 presents a detailed performance statistic,component characteristic and/or performance history for selected firstcomponent at block 1220. The statistic, characteristic and/or historycan pertain to the first component or to a child or children of thefirst components. A performance statistic can include a recent orreal-time performance statistic (e.g., average CPU usage). A componentcharacteristic can include, e.g., resources assigned to the component orequipment of the component. A performance history can include a pastperformance statistic. In some instances, a statistic and/or performancehistory is presented with a threshold value or a comparison (e.g.,population) value. The presentation can include a numerical, text and/orgraphical presentation. For example, performance history can be shown ina line graph. In some instances, different statistics, characteristicsand/or performance history is presented based on a selectioncharacteristic. For example, hovering over a component node can cause anoverall performance statistic for the component to be shown, while moredetailed statistics and/or structure characteristics can be presentedresponsive to a clicking on the component node.

Also responsive to the reviewer's selection, interface engine 375presents identifications of one or more second architecture componentsrelated to the first architecture component at block 1225. Thisidentification can include expanding a representation of thearchitecture to include representations of the second components (whichmay have been previously hidden). In some instances, part of thearchitecture that was initially presented is also hidden at block 1225.This can include, e.g., nodes of components along a non-selected branchin a family-tree architecture. The second components can includecomponents that are children of the first architecture component. Statesassigned to the second architecture components can also be (e.g.,simultaneously) presented.

Interface engine 375 detects a reviewer's selection of one of theidentified second architecture components at block 1230. The selectioncan include a same or similar type of selection as that detected atblock 1215.

Interface engine 375 presents a detailed performance statistic,component characteristic and/or performance history for the selectedsecond component at block 1235. The presentation at block 1235 canmirror that at block 1220 or can be different. In some instances, thepresentation at block 1220 relates to performances and/orcharacteristics of child components of the first component, and thepresentation at block 1235 relates to a performance and/orcharacteristic of the second component (e.g., as the second componentmay not have child components).

FIG. 13 illustrates a flowchart of an embodiment of a process 1300 foranalyzing the performance of a Hypervisor using historical data. Process1300 begins at block 1305, where activity monitor 315 stores thedetected performance metrics in activity data store 320. Block 1305 canparallel block 910 from process 900. Interface engine 375 detects inputfrom a reviewer 125 at block 1310. The input can identify a time period.Identification of the time period can include identifying a duration ofthe time period and/or identifying one or both endpoints of the timeperiod. Identification of an endpoint can include identifying anabsolute date and/or time (e.g., Apr. 1, 2013, 1 pm) or a relative dateand/or time (14 days ago). The input can include a discretization thatcan be used to define discrete time intervals within the time period.The input can include entry of a number and/or text and/or selection ofan option (e.g. using a scroll-down menu, a sliding cursor bar, listmenu options, etc.).

In some instances, a beginning and/or end endpoint of the time periodcan be at least 1, 2, 3, 7, 14, or 21 days or 1, 2, 3, 6, or 12 monthsprior to the detection of the input. The time period can have a durationthat is at least, that is, or that is less than, 1, 4, 8 12 or 24 hours;1, 2, or 4 weeks or 1, 2 or 3 months. Time periods for intra-time-periodtime intervals can be equal to or less than 1, 5, 15 or 30 seconds; 1,5, 15 or 30 minutes; or 1, 2, 4 or 6 hours. The time period could be anytime period going back as far as when performance measurements startedto be collected.

Architecture manager 335 identifies an applicable architecture at block1315. The architecture can be one that characterized a structure of theHypervisor during the identified time period. In some instances, thearchitecture differs from a current architecture. The architecture canbe explicitly or implicitly identified. As an example of implicitidentification, activity data store 320 can index performance metricsaccording to direct and indirect components. Thus, a VM CPU usage can beassociated with both an identifier of the respective VM and anidentifier of a host connected to the VM at the time that the metric wasobtained.

Process 1300 continues then to perform blocks 1320-1330 or 1325-1330 foreach of one, more or all components in the architecture. In instances inwhich the time period is to be analyzed in a discretized manner, blocks1320-1330 or 1325-1330 can also be repeated for each discrete timeinterval in the time period. In these latter cases, it will beappreciated that multiple applicable architectures can be identified toaccount for any architecture changes during the time period.

Statistics generator 340 generates a historical statistic at block 1320.The historical statistic can be of a type similar or the same as aperformance statistic described herein and can be determined in asimilar manner as described herein. It will thus be appreciated that,e.g., depending on a component type, a historical statistic can bedetermined directly based on the performance metrics (e.g., to determinean average CPU usage) or can be determined based on lower-levelcomponent states (e.g., to determine a percentage of VMs withwarning-level CPU usages).

State engine 350 accesses an appropriate state criterion and evaluatesthe generated statistic in view of the criterion. Based on theevaluation, state engine 350 assigns a historical state to the componentat block 1330. Interface engine 375 presents historical performanceindicator(s). The historical indicators can include historicalstatistics and/or historical states. As before, the performanceindicators can be simultaneously presented along with a representationof the applicable architecture (e.g., by distinguishing appearances ofnodes in an architecture family tree based on their states).

Thus, granular low-level performance data can be dynamically accessedand analyzed based on performance characteristics and time periods ofinterest to a reviewer 125. By scanning through time periods, reviewer125 may be able to identify time points at which performance changed.Reviewer 125 can then drill down into the component details tounderstand potential reasons for the change or note any time-lockedarchitecture. Simultaneous presentation of performance indicators andarchitecture representations aid in the ability to detect temporalcoincidence of architecture changes and performance changes.

As noted above, tasks assigned to components can include defining,storing, retrieving and/or processing events. Techniques describedherein can then be used to gain an understanding about whether tasks canbe defined and/or assigned in a different manner which would improvesuch operation (e.g., improve an overall efficiency or improve anefficiency pertaining to a particular type of event). Techniques canfurther be used to identify types of events that generally result inpoor performance or that result in poor performance when assigned toparticular components (or component types) in an information technologyenvironment. Events involved in the tasks can include a variety of typesof events, including those generated and used in SPLUNK® ENTERPRISE.Further details of underlying architecture of SPLUNK® ENTERPRISE are nowprovided.

FIG. 14 shows a block diagram of SPLUNK® ENTERPRISE's data intake andquery system 1400. Data intake and query system 1400 can includeHypervisor components (e.g., a forwarder 1410 or indexer 1415), whichare assigned tasks and monitored, as described in greater detail herein.For example, forwarders 1410 can be assigned data-collection tasks;indexers 1415 can be assigned tasks for segmenting collected data intotime stamped data events, storing the data events in a time-series eventdata store, retrieving select events (e.g., data events, performanceevents, task events and/or structure events) and/or processing retrievedevents. It will therefore be appreciated that the components identifiedin system 1400 are given a functional name. In some exemplary instances,distinct components are defined as forwarders and others as indexers.Nevertheless, in some instances, components are not rigidly functionallydefined, such that a single component may be assigned two or more ofdata-collecting, indexing or retrieval tasks.

Generally, system 1400 includes one or more forwarders 1410 that collectdata from a variety of different data sources 1405, which can includeone or more hosts, host clusters, and/or VMs discussed above, andforwards the data to one or more indexers 1415. The data typicallyincludes streams of time-series data. Time-series data refers to anydata that can be segmented such that each segment can be associated witha time stamp. The data can be structured, unstructured, orsemi-structured and can come from files and directories. Unstructureddata is data that is not organized to facilitate the extraction ofvalues for fields from the data, as is often the case with machine dataand web logs, two popular data sources for SPLUNK® ENTERPRISE.

Tasks defined to a given forwarder can therefore identify a data source,a source type and/or a collection time. In some instances, tasks canfurther instruct a forwarder to tag collected data with metadata (e.g.,identifying a source and/or source-type, such as the one or more hosts145 and VMs 150 discussed above) and/or to compress the data.

Tasks can also relate to indexing of accessible (e.g., collected orreceived) data, which can be performed by one or more indexers 1415.FIG. 15 is a flowchart of a process that indexers 1415 may use toprocess, index, and store data received from the forwarders 1410. Atblock 1505, an indexer 1415 receives data (e.g., from a forwarder 1410).At block 1510, the data is segmented into data events. The data eventscan be broken at event boundaries, which can include charactercombinations and/or line breaks. In some instances, event boundaries arediscovered automatically by the software, and in other instances, theymay be configured by the user.

A time stamp is determined for each data event at block 1515. The timestamp can be determined by extracting the time from data in the dataevent or by interpolating the time based on time stamps from other dataevents. In alternative embodiments, a time stamp may be determined fromthe time the data was received or generated. The time stamp isassociated with each data event at block 1520. For example, the timestamp may be stored as metadata for the data event.

At block 1525, the data included in a given data event may betransformed. Such a transformation can include such things as removingpart of a data event (e.g., a portion used to define event boundaries)or removing redundant portions of an event. A client may specify aportion to remove using a regular expression or any similar method.

Optionally, a key word index can be built to facilitate fast keywordsearching of data events. To build such an index, in block 1530, a setof keywords contained in the data events is identified. At block 1535,each identified keyword is included in an index, which associates witheach stored keyword pointers to each data event containing that keyword(or locations within data events where that keyword is found). When akeyword-based query is received by an indexer, the indexer may thenconsult this index to quickly find those data events containing thekeyword without having to examine again each individual event, therebygreatly accelerating keyword searches.

Data events are stored in an event data store at block 1540. The eventdata store can be the same as or different than a task data store,performance data store and/or structure data store. The data can bestored in working, short-term and/or long-term memory in a mannerretrievable by query. The time stamp may be stored along with each eventto help optimize searching the events by time range.

In some instances, the event data store includes a plurality ofindividual storage buckets, each corresponding to a time range. A dataevent can then be stored in a bucket associated with a time rangeinclusive of the event's time stamp. This not only optimizes time basedsearches, but it can allow events with recent time stamps that may havea higher likelihood of being accessed to be stored at preferable memorylocations that lend to quicker subsequent retrieval (such as flashmemory instead of hard-drive memory).

As shown in FIG. 14, event data stores 1420 may be distributed acrossmultiple indexers, each responsible for storing and searching a subsetof the events generated by the system. By distributing the time-basedbuckets among them, they can find events responsive to a query inparallel using map-reduce techniques, each returning their partialresponses to the query to a search head that combines the resultstogether to answer the query. It will be appreciated that task events,performance events and/or structure events can also be stored in thesame or different time-series data stores that are accessible to each ofmultiple indexers. Thus, queries pertaining to a variety of types ofevents (or combinations thereof) can be efficiently performed. Thisquery handling is illustrated in FIG. 16.

At block 1605, a search head receives a query from a search engine. Thequery can include an automatic query (e.g., periodically executed toevaluate performance) or a query triggered based on input. The query caninclude an identification of a time period, a constraint (e.g.,constraining which events are to be processed for the query, where theconstraint can include a field value), and/or a variable of interest(e.g., a field and/or a statistic type). The query can pertain to asingle type of event or multiple types of events. For example, a querymay request a list of structure characteristics of an environment (e.g.,number of VMs in a Hypervisor) during time periods of strong high-levelperformance (e.g., a minimum VM performance statistic above athreshold). As another example, a query can request data events indexedby a component during an hour of poorest performance over the last 24hours. Processing this request can then include retrieving and analyzingperformance events (to identify the poor-performance hour), task events(to identify tasks performed by the component in the hour), and the dataevents indexed according to the identified tasks. As another example, anautomatic query that routinely evaluates performance correlations canrequest that structure events be evaluated to detect structure changesand that performance events be analyzed to determine any effect that thechanges had on performance.

At block 1610, the search head distributes the query to one or moredistributed indexers. These indexers can include those with access toevent data stores, performance data stores and/or structure data storeshaving events responsive to the query. For example, the indexers caninclude those with access to events with time stamps within part or allof a time period identified in the query.

At block 1615, one or more indexers to which the query was distributedsearches its data store for events responsive to the query. To determineevents responsive to the query, a searching indexer finds eventsspecified by the criteria in the query. Initially, a searching indexercan identify time buckets corresponding to a time period for the query.The searching indexer can then search for events within the buckets forthose that, e.g., have particular keywords or contain a specified valueor values for a specified field or fields (because this employs alate-binding schema, extraction of values from events to determine thosethat meet the specified criteria occurs at the time this query isprocessed). For example, the searching indexer can search forperformance events with performance data corresponding to a particularhost (e.g., by searching for an identifier of the host) or search forweblog events with an identifier of a particular user device.

It should be appreciated that, to achieve high availability and toprovide for disaster recovery, events may be replicated in multipleevent data stores, in which case indexers with access to the redundantevents would not respond to the query by processing the redundantevents. The indexers may either stream the relevant events back to thesearch head or use the events to calculate a partial result responsiveto the query and send the partial result back to the search head.

At block 1620, the search head combines all the partial results orevents received from the parallel processing together to determine afinal result responsive to the query. In some instances, processing isperformed, which can include extracting values of one or more particularfields corresponding to the query, analyzing the values (e.g., todetermine a statistic for a field or to determine a relationship betweenfields).

A query result can be displayed to a reviewer. The query result caninclude extracted values from retrieved events, full retrieved events, asummary variable based on extracted values from retrieved events (e.g.,a statistic, correlation result or model parameter) and/or a graphic(e.g., depicting a change in extracted field values over time orcorrespondences between values of one field and values of another field.In some instances, the display is interactive, such that more detailedinformation is iteratively presented in response to inputs. For example,a first performance indicator for a component can be presented. Aselection input can cause information identifying a number of indexingevents performed by the component during a time period. A further inputcan cause extracted values from indexed events to be presented. Afurther input can cause the events themselves to be presented.

One or more of the blocks in process 1500 and/or process 1600 caninclude an action defined in a task. The task can include appropriateinformation. For example, a task can indicate how events are to betransformed or whether keywords are to be identified or a keyword indexis to be updated. As another example, a task can include a time period(e.g., such that a data-indexing or event-retrieving effort can bedivided amongst indexers).

Data intake and query system 1400 and the processes described withrespect to FIGS. 14-16 are further discussed and elaborated upon inCarasso, David. Exploring Splunk Search Processing Language (SPL) Primerand Cookbook. New York: CITO Research, 2012 and in Ledion Bitincka,Archana Ganapathi, Stephen Sorkin, and Steve Zhang. Optimizing dataanalysis with a semi-structured time series data store. In SLAML, 2010.Each of these references is hereby incorporated by reference in itsentirety for all purposes.

Disclosures herein can therefore enable reviewers to directly reviewcurrent or historical performance data, to view performance dataconcurrently with other data (e.g., characteristics of a structure of acorresponding environment or characteristics of data indexed at a timecorresponding to the performance data) and/or to identify relationshipsbetween types of information (e.g., determining which tasks, taskassignments or structure characteristics are associated with strongperformance). Based on a user-entered time range, it may also bepossible to correlate performance measurements in the time range for aperformance metric with log data from that same time range (where thelog data and/or the performance measurements may both be stored in theform of time-stamped events).

SPLUNK® ENTERPRISE can accelerate queries building on overlapping data,by generating intermediate summaries of select events that can then beused in place of again retrieving and processing the events when thesame query is repeatedly run but later repeats include newer events aswell as the older events. This can be particularly useful whenperformance data is routinely evaluated (e.g., alone or in combinationwith other data types). For example, a query can be generated forrepeated execution. To perform this acceleration, a summary of dataresponsive to a query can be periodically generated. The summaries cancorrespond to defined, non-overlapping time periods covered by thereport. The summaries may (or may not) pertain to a particular query.For example, where the query is meant to identify events meetingspecified criteria, a summary for a given time period may include (ormay identify or may identify timepoints for) only those events meetingthe criteria. Likewise, if the query is for a statistic calculated fromevents, such as the number of events meeting certain criteria, then asummary for a given time period may be the number of events in thatperiod meeting the criteria.

New execution of a query identifying a query time period (e.g., last 24hours) can then build on summaries associated with summary time periodsfully or partly within the query time period. This processing can savethe work of having to re-run the query on a time period for which asummary was generated, so only the newer data needs to be accounted for.Summaries of historical time periods may also be accumulated to save thework of re-running the query on each historical time period whenever thereport is updated. Such summaries can be created for all queries or asubset of queries (e.g., those that are scheduled for multipleexecution). A determination can be automatically made from a query as towhether generation of updated reports can be accelerated by creatingintermediate summaries for past time periods. If it can, then at a givenexecution of a query, appropriate events can be retrieved and fieldvalues can be extracted. One or more intermediate summaries (associatedwith a time period not overlapping with another corresponding summary)can be created and stored.

At each subsequent execution of the query (or execution of another querybuilding on the same data), a determination can be made as to whetherintermediate summaries have been generated covering parts of the timeperiod covered by the current query execution. If such summaries exist,then a query response is based on the information from the summaries;optionally, if additional data has been received that has not yet beensummarized but that is required to generate a complete result, then thequery is run on this data and, together with the data from theintermediate summaries, the updated current report is generated. Thisprocess repeats each time a query using overlapping event datasummarized in a summary is performed. This report acceleration method isused by SPLUNK® ENTERPRISE. It is also described in U.S. patentapplication Ser. No. 13/037,279, which is hereby incorporated byreference in its entirety for all purposes.

FIG. 17 is a flow chart showing how to accelerate automatically queryprocessing using intermediate summaries. At block 1705, a query isreceived. The query can include one generated based on reviewer input orautomatically performed. For example, a query can be repeatedlyperformed to evaluate recent performance of a Hypervisor. The query mayinclude a specification of an absolute time period (e.g., Jan. 5,2013-Jan. 12, 2013) or relative time period (e.g., last week). The querycan include, e.g., specification of a component of interest (e.g., VM#5), a component type of interest (e.g., host), a relationship ofinterest (e.g., number of child VMs supported by a single host) and/or aperformance variable of interest (e.g., component-specifictask-completion latency, average memory usage).

A time period for the query can be identified at block 1710. This timeperiod can include an absolute time period, with a start and end timeand date of the query. A determination can be made at block 1715 as towhether an intermediate summary applicable to the query exists for thequery time period. Stored intermediate summaries can be scanned toidentify those that are associated with summary time periods partly (orfully) within the query time period. Further, selection can berestricted to match data types pertinent to the query. For example, whena query relates purely to performance data, intermediate summariesrelating only to structure data can be avoided.

When it is determined that there is not a pertinent intermediate summaryassociated with a summary time range that includes a portion (e.g., anyportion or a new portion) of the query time range, process 1700continues to block 1720 where new events pertaining to the query areretrieved from one or more data stores. At block 1725, a query result isgenerated using the events. In some situations, one or more intermediatesummaries of retrieved events are generated at block 1730. Each summarycan be associated with a summary time period (e.g., defined based ontime stamps of the events), event type (e.g., performance, structure,data or task) and/or variable type (e.g., a type of performancevariable).

When it is determined that one or more intermediate summaries exist thatsummarize query-pertinent data and that are associated with a summarytime range that includes a summary time range that includes a portion(e.g., any portion or new portion) of the query time range, process 1700continues to block 1735, where those identified summaries are collected.At block 1740, any new events not summarized in a collected summary yetpertinent to the query are retrieved. Information from the collected oneor more summaries can be combined with information from the new eventsat block 1745. For example, values can be extracted from the new eventsand combined with values identified in the intermediate summary. A queryresult (e.g., including a population statistic, relationship or graph)can be generated using the grouped information at block 1750. Process1700 can then continue to block 1730 to generate one or moreintermediate summaries based on the new events.

It will be appreciated that process 1700 may be modified to omit blocks1740, 1745 and 1730. This modification may be appropriate when existingsummaries are sufficient for generating a complete and responsive queryresult.

An acceleration technique that can be used in addition to or instead ofintermediate summaries is use of a lexicon. For each of one or morefields, a lexicon can identify the field, can identify one or morevalues for the field, and can identify (and/or point to) one or moreevents having each of the identified values for the field. Thus, forexample, a first query execution can result in retrieval of a first setof events. Values for one or more fields (e.g., a performance metric)can be extracted from the events (e.g., using a learned or definedlate-binding schema). A lexicon can be generated, accessed and/ormodified that includes a set of values inclusive of the field values.The values in the lexicon can be a single number, a list of numbers or arange of numbers.

For each retrieved event, a representation of the event can be added tothe lexicon. The representation can include an identifier, a pointer tothe event, or an anonymous count increment. The lexicon can beassociated with a time period that includes time stamps of eventscontributing to the lexicon. A lexicon may also or alternatively containa set of keywords (or tokens) and pointers to events that contain thosekeywords. This enables fast keyword searching.

As described with reference to intermediate summaries, intermediatelexicons can be generated for non-overlapping time periods. Subsequentqueries can then use and/or build on lexicons with relevant data togenerate a result. For example, a number of events associated with agiven lexicon value can be counted, an average field value can bedetermined or estimated (e.g., based on counts across multiple lexiconvalues), or correlations between multiple fields can be determined(e.g., since entries for multiple lexicon values can identify a singleevent). In one instance, correlations can also be determined based ondata in multiple lexicons. For example, each point in a set of pointsanalyzed for a correlation or model analysis can correspond to a lexiconand can represent frequencies of values of multiple fields in thelexicon (e.g., a first lexicon having an average value of X1 for fieldF1 and an average value of Y1 for field F2, and a second lexicon havingan average value of X2 for field F1 and an average value of Y2 for fieldF2). U.S. application Ser. No. 13/475,798, filed on May 18, 2012provides additional detail relating to lexicon, and the application ishereby incorporated by reference for all purposes.

Another acceleration technique that can be used in addition to orinstead of intermediate summaries and/or a lexicon is a high performanceanalytics store, which may take the form of data model acceleration(i.e., automatically adding any fields in a data model into the highperformance analytics store). Data model acceleration thus allows forthe acceleration of all of the fields defined in a data model. When adata model is accelerated, any pivot or report generated by that datamodel may be completed much quicker than it would without theacceleration, even if the data model represents a significantly largedataset.

Two exemplary types of data model acceleration may include: ad hoc andpersistent data model acceleration. Ad hoc acceleration may be appliedto a single object, run over all time, and exist for the duration of agiven session. By contrast, persistent acceleration may be turned on byan administrator, operate in the background, and scoped to shorter timeranges, such as a week or a month. Persistent acceleration may be usedany time a search is run against an object in an acceleration-enableddata model.

Data model acceleration makes use of SPLUNK® ENTERPRISE's highperformance analytics store (HPAS) technology, which builds summariesalongside the buckets in indexes. Also, like report accelerationdiscussed above, persistent data model acceleration is easy to enable byselecting a data model to accelerate and selecting a summary time range.A summary is then built that spans the indicated time range. When thesummary is complete, any pivot, report, or dashboard panel that uses anaccelerated data model object will run against the summary rather thanthe full array of raw data whenever possible. Thus, the result returntime may be improved significantly.

Data model acceleration summaries take the form of a time-series index.Each data model acceleration summary contains records of the indexedfields in the selected dataset and all of the index locations of thosefields. These data model acceleration summaries make up the highperformance analytics store. Collectively, these summaries are optimizedto accelerate a range of analytical searches involving a specific set offields—the set of fields defined as attributes in the accelerated datamodel. FIG. 18 is a flow chart showing an exemplary process 1800 forcorrelating performance measurements/values of one or more of theperformance metrics mentioned above of one or more hosts, host clusters,and/or VMs with machine data from the one or more hosts, host clusters,and/or VMs. Process 1800 begins at block 1805 where a set of performancemeasurements (i.e., values of one or more of the above-mentionedperformance metrics) of one or more components of the IT environment arestored, as discussed above, for example, in regards to FIGS. 9A and 9B.The one or more components of the IT environment may include one or moreof each of a host, a cluster, and/or a virtual machine (“VM”). Theperformance measurements may be obtained through an applicationprogramming interface (API) before being stored. The performancemeasurements may be determined by directly observing the performance ofa component, or the performance measurements may be determined throughany of the above-mentioned methods of monitoring performancemeasurements. Further, it is possible for the performance measurementsto be determined without any reference (direct or indirect) to log data.

At block 1810, for each of the performance measurements in the set ofperformance measurements, a time at which the performance measurementwas obtained (or a time to which the performance measurement relates) isassociated with the performance measurement. Each performancemeasurement may be stored in any searchable manner, including as asearchable performance event associated with a time stamp. The timestamp for the performance event may be the associated time at which theperformance measurement was obtained.

Process 1800 continues on to block 1815, in which portions of log dataproduced by the IT environment are stored. For each portion of log data,a time is associated with that portion. This block is similar to theprocess as discussed above in regards to FIG. 15. Each of the portionsof log data may be stored as a searchable event associated with a timestamp. The time stamp for the event that includes the portion of logdata may be the associated time for that portion of log data.

At block 1820, a graphical user interface is provided to enable theselection of a time range. (See FIGS. 19A-19F below). Then, at block1825, through the graphical user interface, a selection of the timerange is received. Optionally, the graphical user interface may allow aselection of a type of performance measurement to be retrieved at block1830. If a selection of a type of performance measurement is received,only the one or more performance measurements of the selected type areretrieved.

The process 1800 then proceeds to block 1835 where one or moreperformance measurements of the set of performance measures stored atblock 1805 are retrieved. Each of the performance measurements that areretrieved has an associated time that is within the selected time rangereceived at block 1825. Also, if optional block 1830 is performed, eachof the one or more performance measurements includes the performancemeasurement of the selected type. At block 1840, one or more portions oflog data stored at block 1810 are retrieved. Each of these retrievedportions of log data has an associated time that is within the selectedtime range received at block 1825.

The retrieved one or more performance measurements and the retrieved oneor more portions of log data may relate to the same host. The retrievedone or more performance measurements may relate to a cluster and the oneor more portions of log data may relate to a host in the cluster.Further, the retrieved one or more performance measurements may relateto a virtual machine and the one or more portions of log data may relateto a host on which that virtual machine has run. A graphical userinterface may be provided to allow a selection of a component. If acomponent is selected, the retrieved one or more performancemeasurements and the retrieved one or more portions of log data mayrelate to the same selected component.

Once the one or more performance measurements and one or more portionsof log data are retrieved, the process proceeds to block 1845 where anindication is displayed for the retrieved performance measurements thathave associated times within the selected time range. At block 1850, anindication of the retrieved portions of log data that have associatedtimes within the selected time range is displayed. The displayedindication of the retrieved performance measurements may be displayedconcurrently with the displaying of the indication of the retrievedportions of log data. Alternatively, the displayed indication of theretrieved performance measurements may be displayed at a different timethan the displaying of the indication of the retrieved portions of logdata. Further, it is possible to display the indication of the retrievedperformance measurements in a same window as the indication of theretrieved portions of log data. (See FIGS. 20A and 20B below). It isalso possible to display the indication of the retrieved performancemeasurements in a different window than the indication of the retrievedportions of log data.

FIGS. 19A-19F illustrates examples of a graphical user interface thatenables the selection of a time range as discussed above with respect toblock 1820 of FIG. 18. FIG. 19A illustrates the selection of a presettime period. As shown in FIG. 19A, preset time periods that can beselected include: the last 15 minutes, the last 30 minutes, the last 60minutes, the last 4 hours, the last 24 hours, the last 7 days, the last30 days, last year, today, week to date, business week to date, month todate, year to date, yesterday, previous week, previous business week,previous month, previous year, and all time (since when performancemeasurements were first obtained and stored). Also shown in FIG. 19A isthe corresponding display of an indication of the retrieved performancemeasurements that have associated times within the selected time rangeof block 1845, and indication of the retrieved portions of log data thathave associated times within the selected time range of block 1850.

As shown in FIG. 19B, a reviewer, such as reviewer 125, can select acustom time range. When a custom time setting is selected, a custom timerange visualization may be presented, as shown in FIGS. 19C-19F. Thecustom time range visualization allows a reviewer to enter an earliestdate for data of a report and a latest date for the data of the reportthrough a variety of methods. A reviewer may enter actual dates to beused to generate the report, a relative time period to be used togenerate the report, a time window for which the report is to providereal-time data, and may enter a custom time range by using a searchlanguage.

FIG. 19C illustrates one embodiment that allows the reviewer to generatea report by entering a time period by using a search language, such asSplunk Search Processing Language (SPL), as discussed above. Thereviewer may enter an earliest time period of the report and a latesttime period of the report in the search language, and the custom timerange visualization may present the reviewer with the actual dates forthe earliest date and latest date. The report will be generated from theentered search language.

As shown in FIG. 19D, a reviewer may request a real-time report that isgenerated based on the time window entered by the reviewer. The timewindow entered could be any number of seconds, minutes, hours, days,weeks, months, and/or years. Once the time window is entered, the customtime range visualization may present the reviewer with a search languageequivalent of the time window requested. The report will be generatedfrom the time window entered by the reviewer.

A reviewer may also enter a relative time range to generate a report asshown in FIG. 19E. In this embodiment, the reviewer would enter theearliest time desired for the report. The earliest time entered could beany number of seconds, minutes, hours, days, weeks, months, and/or yearsago, and the latest time period would be the present. Once the timewindow is entered, the custom time range visualization may present thereviewer with a search language equivalent of the time range requested.The report will be generated from the relative time range entered by thereviewer.

FIG. 19F illustrates a custom time range visualization that allows areviewer to enter an earliest time for a time range of the report and alatest time of a time range of the report directly. The reviewer mayenter a specific earliest time for the report or request the earliesttime to be the earliest date of the data available. The reviewer mayalso enter the specific latest time for the report or request to use thepresent. Once entered, the report will be generated based on the timesentered.

FIGS. 20A and 20B illustrate a display of an indication of a retrievedperformance measurements in a same window as an indication of theretrieved portions of log data. In alternative embodiments, theinformation about the performance measurements and the information aboutthe log data could be displayed in separate windows or could bedisplayed sequentially rather than concurrently. FIG. 20A illustrates anexample where the performance measurements of the set of performancemeasurements is an average CPU core utilization percent metric. Each ofthe performance measurements that are retrieved has an associated timethat is within the selected time range received. Of course, theperformance measurement may be any of the above-mentioned performancemetrics. FIG. 20B illustrates an example where the graphical userinterface may allow a selection of a type of performance measurement tobe retrieved at block 1830 of FIG. 18. If a selection of a type ofperformance measurement is received, only the one or more performancemeasurements of the selected type are retrieved.

From the display of an indication of a retrieved performancemeasurements with an indication of the retrieved portions of log data, areviewer may interact with the display to retrieve the raw log dataassociated with the portions of log data and performance measurements,as shown in FIG. 21. This allows a reviewer to easily access and viewevents directly.

hence, as disclosed above, methods and computer-program products areprovided for storing a set of performance measurements relating toperformance of a component in an IT environment, and associating withthe performance measurement a time at which the performance measurementwas obtained for each performance measurement in the set of performancemeasurements. The methods and computer-program products include storingportions of log data produced by the IT environment, wherein eachportion of log data has an associated time; providing a graphical userinterface enabling selection of a time range; and receiving through thegraphical user interface a selection of a time range. The methods andcomputer-program products further comprise retrieving one or moreperformance measurements, wherein each of the retrieved performancemeasurements has an associated time in the selected time range;retrieving one or more portions of log data, wherein each of theretrieved portions of log data has an associated time in the selectedtime range; displaying an indication of the retrieved performancemeasurements having their associated times in the selected time range;and displaying an indication of the retrieved portions of log datahaving their associated times in the selected time range.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, or in computer software, firmware, or hardware, including thestructures disclosed in this specification and their structuralequivalents, or in combinations of one or more of them. Embodiments ofthe subject matter described in this specification can be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.

The computer readable medium can be a machine readable storage device, amachine readable storage substrate, a memory device, a composition ofmatter effecting a machine readable propagated signal, or a combinationof one or more of them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus can include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a data store management system, an operating system, ora combination of one or more of them, A propagated signal is anartificially generated signal, e.g., a machine generated electrical,optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code), can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., on or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnonvolatile memory, media, and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, architecture provider orreviewer, embodiments of the subject matter described in thisspecification can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube) to LCD (liquid crystal display) monitor,for displaying information to the user and a keyboard and a pointingdevice, e.g., a mouse or a trackball, by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user, architecture provider or reviewer as well; forexample, feedback provided to the user can be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback;and input from the user, architecture provider or reviewer can bereceived in any from, including acoustic, speech, or tactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context or separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the invention have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results.

What is claimed is:
 1. A method comprising: acquiring, by a computersystem, performance measurements for a performance metric associatedwith one or more hardware or software components of an informationtechnology (IT) environment; acquiring, by the computer system,structure data indicative of structure characteristics of the ITenvironment, the structure data associated with the one or more hardwareor software components of the IT environment; correlating, by thecomputer system, at least a portion of the performance measurements forthe performance metric with at least a portion of the structure dataassociated with the one or more hardware or software components of theIT environment, based on a correlation criteria; and causing display,via a graphical user interface, of an indication of performance of atleast one hardware or software component of the IT environment inassociation with the performance metric.
 2. The method of claim 1,wherein the correlation criterion relates to a machine in the ITenvironment.
 3. The method of claim 1, wherein the structure data arederived from log data associated with the one or more hardware orsoftware components of the IT environment.
 4. The method of claim 1,wherein the structure data are derived from log data associated with theone or more hardware or software components of the IT environment, andwherein the log data is from a text-based log file.
 5. The method ofclaim 1, wherein the structure data are derived from log data associatedwith the one or more hardware or software components of the ITenvironment, the log data is from a text-based log file, and theperformance measurements are not derived from a log file and areacquired independently of the log data.
 6. The method of claim 1,wherein the plurality of performance measurements are acquired withoutreference to log data.
 7. The method of claim 1, wherein the performancemeasurements have been acquired by direct measurement of the one or morehardware or software components of the IT environment.
 8. The method ofclaim 1, wherein the structure data are derived from log data associatedwith the one or more hardware or software components of the ITenvironment; the method further comprising: acquiring the log data inthe form of a plurality of portions of log data; and storing each of theacquired portions of log data.
 9. The method of claim 1, wherein thestructure data are derived from log data associated with the one or morehardware or software components of the IT environment; the methodfurther comprising: acquiring the log data in the form of a plurality ofportions of log data; and wherein said storing comprises storing each ofthe acquired performance measurements, each of the acquired structuredata and each of the acquired portions of log data with a time-stamp ina time-series data store.
 10. The method of claim 1, wherein thecorrelation criterion is a time-based criterion.
 11. The method of claim1, wherein the correlation criterion is not a time-based criterion. 12.The method of claim 1, wherein the correlation criterion is auser-specified search criterion.
 13. The method of claim 1, wherein thecorrelation criterion is a user-specified search criterion that is not atime-based criterion.
 14. The method of claim 1, wherein saidcorrelating comprises causing concurrent display of at least oneperformance measurement and at least one structure data that eachsatisfy the correlation criterion.
 15. The method of claim 1, whereinthe performance metric comprises a performance metric for one or morehardware or software resources of a computer system.
 16. The method ofclaim 1, wherein the performance metric comprises a performance metricfor at least one virtual machine or virtual machine host.
 17. The methodof claim 1, wherein the performance metric comprises a performancemetric for a virtual machine cluster.
 18. The method of claim 1, furthercomprising: obtaining the correlation criterion from a user input to agraphical user interface element that enables a user to input a searchcriterion as said correlation criterion but does not enable input of acomplete search query.
 19. The method of claim 1, further comprising:obtaining the correlation criterion from a user input to a graphicaluser interface element, wherein the correlation criterion relates to amachine in the IT environment.
 20. The method of claim 1, furthercomprising, prior to said correlating: obtaining the correlationcriterion from a search query input by a user into a query box in atext-based search query language.
 21. The method of claim 1, furthercomprising, prior to said correlating: obtaining the correlationcriterion from a search query input by a user into a query box in atext-based search query language, wherein the correlation criterionrelates to a machine in the IT environment.
 22. The method of claim 1further comprising: storing, by the computer system, the acquiredperformance measurements and the acquired structure data, wherein saidstoring comprises storing each of the acquired performance measurementsand each of the acquired structure data with a time-stamp; saidcorrelating further comprises identifying at least one of the storedperformance measurements and at least one of the stored structure datathat have time stamps that satisfy a user-specified time criterion. 23.The method of claim 1 further comprising: storing, by the computersystem, the acquired performance measurements and the acquired structuredata, wherein said storing comprises storing each of the acquiredperformance measurements and each of the acquired structure data with atime-stamp in a time-series data store; and said correlating comprisescorrelating a performance measurement of the IT environment with astructure data of the IT environment, by identifying a portion of storedstructure data that has a time stamp that corresponds to a time stamp ofone of the stored performance measurements.
 24. The method of claim 1further comprising: storing, by the computer system, the acquiredperformance measurements and the acquired structure data, wherein saidstoring comprises storing each of the acquired performance measurementsand each of the acquired structure data with a time-stamp; saidcorrelating comprises: identifying a particular performance measurement;and identifying a portion of structure data that has a time stamp mostrecently preceding a time stamp associated with the particularperformance measurement.
 25. The method of claim 1, wherein theperformance measurements are stored in a first time-series data store,and the structure data are stored in a second time-series data storeseparate from the first time-series data store.
 26. The method of claim1, wherein the performance measurements are stored in a time-series datastore in a first format, and the log data are stored in said time-seriesdata store in a second format different from the first format.
 27. Themethod of claim 1, wherein the performance measurements are stored in afirst time-series data store in a first format, and the structure dataare stored in a second time-series data store separate from the firsttime-series data store in a second format different from the firstformat.
 28. The method of claim 1, the method further comprising:acquiring log data indicative of activity of the one or more hardware orsoftware components of the IT environment; storing each of the acquiredportions of log data with an associated time stamp in a firsttime-series data store; and storing the acquired performancemeasurements and the acquired structure data, wherein each of theacquired performance measurements are stored with an associated timestamp in a second time-series data store separate from the firsttime-series data store, and wherein each of the acquired structure dataare stored with an associated time stamp in a third time-series datastore separate from the first and second time-series data stores.
 29. Anon-transitory machine-readable storage medium for use in a processingsystem of a data intake and query system, the non-transitorymachine-readable storage medium storing instructions, an execution ofwhich in the processing system causes the processing system to performoperations comprising: acquiring, by a computer system performancemeasurements for a performance metric associated with one or morehardware or software components of an information technology (IT)environment; acquiring, by the computer system, structure dataindicative of structure characteristics of the IT environment, thestructure data associated with the one or more hardware or softwarecomponents of the IT environment; correlating, by the computer system,at least a portion of the performance measurements for the performancemetric with at least a portion of the structure data associated with theone or more hardware or software components of the IT environment, basedon a correlation criteria; and causing display, via a graphical userinterface, of an indication of performance of at least one hardware orsoftware component of the IT environment in association with theperformance metric.
 30. A system comprising: a communication devicethrough which to communicate on a computer network; and at least oneprocessor operatively coupled to the communication device and configuredto perform operations including acquiring, by a computer system,performance measurements for a performance metric associated with one ormore hardware or software components of an information technology (IT)environment; acquiring, by the computer system, structure dataindicative of structure characteristics of the IT environment, thestructure data associated with the one or more hardware or softwarecomponents of the IT environment; correlating, by the computer system,at least a portion of the performance measurements for the performancemetric with at least a portion of the structure data associated with theone or more hardware or software components of the IT environment, basedon a correlation criteria; and causing display, via a graphical userinterface, of an indication of performance of at least one hardware orsoftware component of the IT environment in association with theperformance metric.